Inside NVIDIA’s instruction-cache patent: how a small choice about what to fetch next tries to keep a processor moving.
US9110810B2 · Independent PatentOracle research · 20 September 2026
Each prefetch operation chooses one future cache line. Across repeated operations, a pseudorandom value and the current sector position help bring multiple possible next lines into the small, fast L1 cache.
In the worked example, bit 0 at sector 0 selects a target six lines ahead. This reaches the next larger L1.5 line early, allowing more time for a slower retrieval from L2 or memory.
More look-ahead can help hide retrieval delays. A branch can break the expected sequence. The patent proposes varying the target rather than always reaching the same distance ahead; it supplies no measured speed-up.
This 2015 disclosure addresses instruction fetching through a hierarchy in which the instruction L1 cache is fast but limited, an L1.5 cache is an intermediate source, and L2 is a farther fallback. Its central technique is not simply to prefetch the immediately next line. On an L1 hit, fetch unit 410 uses both the current L1 line's sector position in L1.5 cache 335 and a random or pseudo-random value to select a future L1-sized line as a prefetch target. The intended effect is to keep several plausible future instruction lines available in L1 while varying the look-ahead distance.
Sequential instruction execution makes prefetching useful: a line moved into L1 before demand can become an L1 hit. The background identifies the opposing risk. A fast processor may need more than one line of look-ahead to hide latency, but a fixed policy that prefetches two or more lines ahead can be harmful after a branch to a non-sequential target: several anticipated lines may be irrelevant, while the target can incur multiple misses. The disclosure presents probabilistic, position-sensitive target selection as a way to balance these concerns. It states improved L1 hit rate and reduced instruction-fetch latency as advantages; the supplied material does not report benchmark workloads, measurements, percentages, or comparisons. Those are stated expected advantages, not measured results.
Original FIG. 4. Blue numbered leaders identify parts; the patent’s black connections and labels are unchanged.
Inspect original page 7 ↗The fetch unit brings cache lines into L1 and L1.5 from higher levels or system memory. It places a missing instruction line in L1 so the processor can execute it.
Instructions accessed by processors in SM 310 are supplied here. When a needed line is absent, it must be retrieved from a higher level.
An L1.5 hit supplies the requested L1-sized sector. A miss sends the fetch unit through the MMU toward L2.
L2 is the next source after an L1.5 miss. The description adds a system-memory fallback if L2 also misses.
Original FIG. 4 on page 7, read with its description, identifies instruction L1 cache 370, SM 310, L1.5 cache 335, GPC 208, fetch unit 410, MMU 328, L2 cache 420, and system memory 104 as the multi-level instruction-cache embodiment. Fetch unit 410 fills L1 and L1.5 from higher cache levels or memory. An L1.5 line is divided into sectors, with each sector corresponding to one instruction-L1 line. In the worked example, an L1 line is 64 bytes and contains eight 8-byte instructions; an L1.5 line is 256 bytes and contains 32 instructions, so it has four 64-byte sectors. This is an example embodiment, rather than a universal size requirement.
Original FIG. 5. The description identifies 530–544 as prefetch-target relationships. The offsets below belong to its worked 64-byte-L1 example.
Inspect original page 8 ↗The example L1.5 line contains four sectors. Each sector has the size of one 64-byte instruction-L1 line.
From sector 512(0), the illustrated target 530 is 512(1). The same one-sector progression applies to the other current sectors.
With current sector 512(1) and bit 0, target 540 is 512(3): two sectors, or 128 bytes, ahead.
From 512(0), target 538 reaches 522(2) in the next L1.5 line: six sectors, or 384 bytes, ahead. This starts that next-line retrieval early.
Original FIG. 5 shows L1.5 Cache Line 0, reference 510, above L1.5 Cache Line 1, reference 520. Line 510 contains, top to bottom, sectors 512(0), 512(1), 512(2), and 512(3); line 520 continues with 522(0), 522(1), 522(2), and 522(3). That ordering represents consecutively increasing addresses. The right-side arrows 530, 532, 534, and 536 point to the one-sector-ahead targets. The routed left-side arrows 538, 540, 542, and 544 point to the farther alternatives. The accompanying description identifies these arrows by their prefetch-target relationships in this example.
Original FIG. 6. Keep the YES and NO branches in view: a demand miss and a speculative target use different entry paths into the same higher-cache retrieval route.
Inspect original page 9 ↗YES bypasses prefetch selection and goes to 620. The fetch unit needs to retrieve the current demand from a higher cache.
The NO branch at 608 determines the sector, computes a random-number index and selects a target from both values.
NO ends the method because the target is already in L1. YES continues to the L1.5 check at 620.
At 620, a hit takes the NO route to 622 and retrieves from L1.5. A miss takes the YES route to 624 and retrieves from L2.
FIG. 6 begins at 602, where fetch unit 410 retrieves the program counter, then at 604 determines the fetch target: the instruction-L1 line containing that address. At 606 it attempts retrieval from L1. If decision 608 says “YES,” the target is absent from L1 and this is a demand-fetch path, not a prefetch opportunity. Decision 620 tests L1.5: a “NO” goes to 622, “RETRIEVE TARGET FROM L1.5 CACHE,” and a “YES” goes to 624, “RETRIEVE TARGET FROM L2 CACHE.” The broader description says that an L2 miss can in turn lead to system memory. Once found, the relevant L1-sized sector/cache line is transferred into L1 for execution.
If decision 608 says “NO,” the demand line is already in L1. The vertical path then determines the corresponding L1.5 sector at 610, computes a random-number index at 612, and determines a prefetch target at 614. It tests that target in L1 at 616 and decision 618. If 618 says “NO,” the target is already resident and the method ends. If it says “YES,” the target is obtained through the same L1.5/L2 route at 620, 622, and 624. The figure's geometry makes the distinction clear: the “YES” output of demand-miss decision 608 runs around the right side to 620, while the L1-hit route descends through selection steps 610–618.
For the illustrated one-bit example, random bit 1 selects one sector, or 64 bytes, ahead: arrows 530 through 536 successively map 512(0) to 512(1), 512(1) to 512(2), 512(2) to 512(3), and 512(3) to 522(0). With bit 0, sector 512(1) maps through arrow 540 to 512(3), 512(2) maps through 542 to 522(0), and 512(3) maps through 544 to 522(1): two sectors, or +128 bytes, ahead. The exception is current sector 512(0), which maps through 538 to 522(2), six sectors or +384 bytes ahead. The stated purpose of this distant choice is to initiate an early prefetch for the next L1.5 line, whose sector 0 may otherwise require a longer L2 or system-memory retrieval.
Claim 1 recites a computer-implemented method: determine a memory location within a first cache; generate a pseudo-random number; select a first-cache line for prefetch associated with that location and number; and, in response to a prefetch miss, locate a corresponding first sector in a first line of a second cache and transfer that sector to the first-cache line. Its key limitation is that distance between the transferred first sector and the memory location's second sector depends on both the pseudo-random value and the second sector's position in the second cache. Claim 2 depends from claim 1 and narrows the first cache to an instruction cache.
Claim 3, dependent on claim 1, specifies conditional alternatives: one sector for a first pseudo-random value; two sectors for a second value when the second sector is in a first position; and six sectors for that second value when the second sector is in a second position. These are alternatives, not three simultaneous requirements. Claim 4 depends on claim 3 and says the second sector is in a second-cache line different from the first-cache line containing the transferred first sector. Claims 5 and 6 respectively add equal sector/L1-line size and four sectors per second-cache line; claim 6 depends on claim 5. Claim 7 adds a third-cache fallback, and claim 8 adds a common fetch unit, each directly from claim 1. Claims 9 and 17 express related subsystem and computing-device forms, with their own dependent claim chains.
Begin with FIG. 4 on original page 7: locate fetch unit 410, instruction L1 cache 370, L1.5 cache 335 and L2 cache 420. Its description explains how a missing instruction is brought into L1. Then follow FIG. 6 on page 9: demand-miss decision 608 leads to the higher-cache route; its NO branch opens the prefetch selection at 610–618. Use FIG. 5 on page 8 to compare near and distant targets: arrows 530–536 select the next sector, while 538 and 540–544 show the farther choices. In this worked example the offsets are +64, +128 or, from sector 0 with bit 0, +384 bytes. Finally, read claims 1–8 to separate the core method from the dependent instruction-cache, geometry, fallback and common-fetch-unit limitations. Source links open the preserved pages; the matching PDF includes every original page. This is a historical disclosure, not evidence about current NVIDIA GPU internals.
One historical patent, with all ten identified description and claim pages included in the source packet. Three original figures are explained here. All 19 original pages accompany the PDF; the remaining drawings are preserved for inspection, rather than individually annotated.
The PDF contains the explanation, annotations and every original page, with internal source links. The ZIP includes this web edition, the PDF, source and review records, and checksums. Downloaded files remain readable offline.
The automated draft and bounded revision were checked against source passages and selected original images. Codex then resolved four remaining review points: supplemental FIG. 4 coverage, the wording of FIG. 5 arrow roles, the claim 2 citation and the complete reading route. The original AI verdict is preserved. The numbered overlays and final presentation received a separate Codex source and visual inspection; this is not independent human review.
The AI image review covered original pages 8, 9, 17 and 18. FIG. 4 on page 7 was added through recorded assisted source inspection, with its description on pages 15–16.
The byte sizes and one-, two- and six-sector offsets are a worked embodiment. They are not universal requirements for all caches or evidence of how current NVIDIA GPUs operate.
The original patent is the source of the proposed capabilities. This independent example does not establish deployment, measured performance, a client relationship or endorsement by NVIDIA.
Start from this example, change the publication and tell us what matters to you. Review the scope and price before generation begins.
Commission something like this ↗