Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block-cache read Pipeline #1462

Merged
merged 69 commits into from
Aug 9, 2024
Merged
Changes from 16 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
c30ba14
spell check
May 7, 2024
08707ce
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
May 8, 2024
0852969
merge
May 10, 2024
4afd00b
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft May 15, 2024
5006528
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft May 16, 2024
0628e9e
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft May 19, 2024
ed3729e
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jun 19, 2024
fa04d94
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jun 21, 2024
95447ff
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jun 24, 2024
6052556
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jul 10, 2024
c24b669
fix
ashruti-msft Jul 10, 2024
7e5001a
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jul 11, 2024
551b9b1
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jul 15, 2024
e033855
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jul 16, 2024
54ed05b
first
ashruti-msft Jul 17, 2024
e91f289
returns eof if block is not filled
ashruti-msft Jul 17, 2024
009047b
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jul 17, 2024
6e53a10
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jul 23, 2024
962e1c2
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jul 26, 2024
37527f6
corrected check condition for EOF
ashruti-msft Jul 26, 2024
a3d3e65
Test
ashruti-msft Jul 27, 2024
566bf9c
fixe return statement
ashruti-msft Jul 27, 2024
7b13164
fixed name of test
ashruti-msft Jul 27, 2024
be7393f
generaye parquet files
ashruti-msft Jul 27, 2024
b78bfbf
del
ashruti-msft Jul 27, 2024
42d44b1
fix
ashruti-msft Jul 27, 2024
f11ba00
fix
ashruti-msft Jul 27, 2024
8c122d0
install pip3
ashruti-msft Jul 29, 2024
6d5b9ca
Added UT for checking total bytes read
ashruti-msft Jul 29, 2024
9f2e31e
checking disk size
ashruti-msft Jul 29, 2024
e52942e
Added file size check
ashruti-msft Jul 29, 2024
023dd57
fix yaml
ashruti-msft Jul 29, 2024
0b2fe15
checking err in file size check
ashruti-msft Jul 29, 2024
c8f5bba
printing bc details
ashruti-msft Jul 29, 2024
5395f94
change
ashruti-msft Jul 29, 2024
e332be9
added run with diff blocksize
ashruti-msft Jul 29, 2024
4d6da89
fix yaml
ashruti-msft Jul 29, 2024
26831d5
fix yaml
ashruti-msft Jul 29, 2024
b83be5b
changed myfile to datafile
ashruti-msft Jul 29, 2024
8bc4d88
changed myfile to datafile
ashruti-msft Jul 29, 2024
1e08558
changed directory structure and removed unwanted commands
ashruti-msft Jul 29, 2024
08908bd
changed test for file size check
ashruti-msft Jul 29, 2024
43626ea
Clear files on work dir
ashruti-msft Jul 29, 2024
279c256
added check for EOF err and unmount and mount before checking md5sum
ashruti-msft Jul 30, 2024
e69f66f
Merge remote-tracking branch 'origin/main' into ashruti/bcReadCorr
ashruti-msft Jul 30, 2024
d62d7c3
spell check
ashruti-msft Jul 30, 2024
3cb4d42
changed lh to l exiting on fail size check
ashruti-msft Jul 30, 2024
2e51140
test
ashruti-msft Jul 30, 2024
e04d052
test
ashruti-msft Jul 30, 2024
fce85f4
filecache
ashruti-msft Jul 30, 2024
ed14c35
fixyaml
ashruti-msft Jul 30, 2024
52d2289
fixyaml
ashruti-msft Jul 30, 2024
f52bec3
fixyaml
ashruti-msft Jul 30, 2024
51648df
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jul 30, 2024
3040e89
filesizecheck
ashruti-msft Jul 30, 2024
5f22f37
Merge remote-tracking branch 'origin/main' into ashruti/bcReadCorr
ashruti-msft Jul 31, 2024
6333d32
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Jul 31, 2024
7b0fec8
added byte validation
ashruti-msft Jul 31, 2024
cfc27fa
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Aug 1, 2024
7f09150
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Aug 2, 2024
fc9a481
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Aug 5, 2024
23d6002
Merge branch 'main' of https://github.com/Azure/azure-storage-fuse in…
ashruti-msft Aug 5, 2024
7e87244
Merge branch 'main' into ashruti/bcReadCorr
ashruti-msft Aug 5, 2024
ccd6711
removed file size and bytes check as not required if md5sum matches
ashruti-msft Aug 5, 2024
db6e051
add tests with block-cache direct io
ashruti-msft Aug 5, 2024
e1870a8
fix
ashruti-msft Aug 6, 2024
2c027f1
redirecting cat to temp file
ashruti-msft Aug 7, 2024
e25c87a
added disk cache test
ashruti-msft Aug 8, 2024
cb6058c
minor changes
ashruti-msft Aug 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion component/block_cache/block_cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -522,11 +522,15 @@ func (bc *BlockCache) ReadInBuffer(options internal.ReadInBufferOptions) (int, e

// Copy data from this block to user buffer
readOffset := uint64(options.Offset) - block.offset
bytesRead := copy(options.Data[dataRead:], block.data[readOffset:])
bytesRead := copy(options.Data[dataRead:], block.data[readOffset:(block.endIndex-block.offset)])
vibhansa-msft marked this conversation as resolved.
Show resolved Hide resolved

// Move offset forward in case we need to copy more data
options.Offset += int64(bytesRead)
dataRead += bytesRead

if (block.endIndex - block.offset) < uint64(len(block.data)) {
vibhansa-msft marked this conversation as resolved.
Show resolved Hide resolved
return 0, io.EOF
}
}

return dataRead, nil
Expand Down
Loading