Nim text reader #86
Merged
Nim text reader #86
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
September 1, 2023 15:23
… and used for datetime parser
added 24 commits
September 25, 2023 11:06
… all of the tests and not it individually, something to do with shared state? is it a problem with nimporter or nim? i don't think there's any stateful code on my end
…ttempting to deallocate unallocated memory and the traceback is wrong!
|
Great work!
I'll help you with windows on Monday. I have most of the day available.
I'll try to start early Monday morning so you can get me working on it.
I also don't see it as a show stopper as we can release it to dce as a .dev
version
…On Sat, 30 Sept 2023, 13:19 Ratchet, ***@***.***> wrote:
I'm likely going to merge this on Monday, you can see if you want to add
something else.
I will also rebase so that the test pipeline commits don't spam the commit
history.
Known issues: I can't get it to work within Windows pipelines so I'll need
help from someone with windows to get python working with nim.
I will also add a fallback to use the old python implementation as a
fallback if it throws within nim until we're confident with maintaining nim
only implementation.
------------------------------
You can view, comment on, or merge this pull request online at:
#86
Commit Summary
- 3fa91d5
<3fa91d5>
* changes
- 2132317
<2132317>
* ocd
- 223b6c2
<223b6c2>
* missing word
- adf357d
<adf357d>
* headers are supposed to be strings
- 2e2bcd5
<2e2bcd5>
* stuff
- 3357f45
<3357f45>
* seems to work
- 286314e
<286314e>
* dont have parity yet but seems faster
- b7c85da
<b7c85da>
works, no parity in terms of functionality but it's fast
- e3fc29c
<e3fc29c>
cleanup
- f9dcc1e
<f9dcc1e>
* working on parsing stuff
- d8086a3
<d8086a3>
* forgot to save
- ccea962
<ccea962>
* only missing date classes
- c60bcff
<c60bcff>
* working on pickling date
- 9941c9e
<9941c9e>
* dumping tiem
- 1b9994d
<1b9994d>
* rewrote the date parser so that it could be merged with time parser and
used for datetime parser
- b9e6c8f
<b9e6c8f>
* check daye range
- 49fd910
<49fd910>
* i think most of the formats work not to cleanup code and to fix leftover
bugs
- fe04bca
<fe04bca>
* split file encoder from main script
- 4b86543
<4b86543>
* split csv parser from main
- 639f870
<639f870>
* split pickling from main
- a5909b7
<a5909b7>
* split infer types and numpy from main
- 75fa51d
<75fa51d>
* rank update
- 20651f8
<20651f8>
* modular
- 0023e19
<0023e19>
* improve type inference
- cc7a690
<cc7a690>
* cleanup
- a10ba39
<a10ba39>
* fix date
- 8f5964b
<8f5964b>
* fixed date range but need to account for shortdate+int format
- aa32274
<aa32274>
* improve heuristics when short format + int
- 2315b3b
<2315b3b>
*updates
- 18e8512
<18e8512>
* working on dumping back to python
- 4902b97
<4902b97>
* python layer
- f7d0318
<f7d0318>
* pass dtypes, bugfixes left
- b9df065
<b9df065>
* cleanup
- 79277bf
<79277bf>
* cleanup
- 88013a4
<88013a4>
* fix some stuff
- 9327ed1
<9327ed1>
* fixed some stuff
- 4239ed8
<4239ed8>
* fix pylayer
- 06744dc
<06744dc>
* added to pycode
- cd9083b
<cd9083b>
* switching branches because other stuff came up
- 902af67
<902af67>
Merge branch 'master' into nim-csv
- f28d674
<f28d674>
* added start/limit
- 71ea781
<71ea781>
* fix no qualifier
- 8e82c69
<8e82c69>
+ win1251 support
- d61077e
<d61077e>
+ add skip trailing space
- 7081920
<7081920>
+ header row fla support
- 65dd612
<65dd612>
+ header row offset
- 770307a
<770307a>
all but one of the tests pass, and even then it's only when executing all
of the tests and not it individually, something to do with shared state? is
it a problem with nimporter or nim? i don't think there's any stateful code
on my end
- a798a7d
<a798a7d>
* why was the test re-guessing the dtype? and failing?
- 0255666
<0255666>
* undo
- 6d2fd3d
<6d2fd3d>
* parsing int/float now supports tablite locale parsin
- 632bebf
<632bebf>
* fix and optimize utf16 reader
- cc5ae40
<cc5ae40>
* arparse
- 11bcfce
<11bcfce>
* some bugfixes
- 296bde4
<296bde4>
* fix gc, pointer deferencing actually copies but gc seems to do some shady
stuff
- 1bbc097
<1bbc097>
* removed pointers because nim seems to poop itself in gc when they are used
- d4f33e7
<d4f33e7>
* version
- bbf02a3
<bbf02a3>
* fix nim 2.0.0 support, there's a bug in the GC when using notinit
attempting to deallocate unallocated memory and the traceback is wrong!
- 11ceb37
<11ceb37>
* cleanup
- 0686ab7
<0686ab7>
* hmm
- 083a03f
<083a03f>
Merge branch 'nim-csv'
- 92e4a91
<92e4a91>
Test pipelines
- 4ee6f0b
<4ee6f0b>
Test pipelines
- 06d51c9
<06d51c9>
what the
- 8dc0e9a
<8dc0e9a>
i'm stupid
- debd830
<debd830>
Test pipelines, I hate windows edition
- ed10d94
<ed10d94>
Test pipelines, I hate windows edition
- 1b87276
<1b87276>
Test pipelines, I hate windows edition
- 863fa46
<863fa46>
Test pipelines, I hate windows edition
- e24b2d8
<e24b2d8>
Test pipelines, I hate windows edition
- d46785c
<d46785c>
Test pipelines, I hate windows edition
- 62b1319
<62b1319>
Test pipelines, I hate windows edition
- 0157ffa
<0157ffa>
Test pipelines, I hate windows edition
- 7f02543
<7f02543>
Test pipelines, I hate windows edition
- d22d832
<d22d832>
Test pipelines, I hate windows edition
- 2ef7248
<2ef7248>
Test pipelines, I hate windows edition
- 5a2a9ae
<5a2a9ae>
Test pipelines, I hate windows edition
- bbf7d6d
<bbf7d6d>
Test pipelines, I hate windows edition
- 427ef73
<427ef73>
Test pipelines, I hate windows edition
- a246dbb
<a246dbb>
Test pipelines, I hate windows edition
- b6d7e68
<b6d7e68>
Test pipelines, I hate windows edition
- cd3c06e
<cd3c06e>
Test pipelines, I hate windows edition
- ab3c5bd
<ab3c5bd>
Test pipelines, I hate windows edition
- b687cbc
<b687cbc>
Test pipelines, I hate windows edition
- 3383689
<3383689>
Test pipelines, I hate windows edition
- c691ec6
<c691ec6>
Test pipelines, I hate windows edition
- ad875d9
<ad875d9>
Test pipelines, I hate windows edition
- 2faf3d4
<2faf3d4>
Test pipelines, I hate windows edition
- 623b44a
<623b44a>
Test pipelines, I hate windows edition
- 4a55c29
<4a55c29>
Test pipelines, I hate windows edition
- ea2d726
<ea2d726>
+ inlining
- 787761e
<787761e>
Update ranking.nim
File Changes
(32 files <https://github.com/root-11/tablite/pull/86/files>)
- *M* .github/workflows/publish.yml
<https://github.com/root-11/tablite/pull/86/files#diff-551d1fcf87f78cc3bc18a7b332a4dc5d8773a512062df881c5aba28a6f5c48d7>
(62)
- *M* .github/workflows/python-test.yml
<https://github.com/root-11/tablite/pull/86/files#diff-0165ee7fe5057e5c6e9f2ae4ae3d87d21324f691385cbf51b220a4abe79ce492>
(40)
- *M* .gitignore
<https://github.com/root-11/tablite/pull/86/files#diff-bc37d034bad564583790a46f19d807abfe519c5671395fd494d8cce506c42947>
(2)
- *A* _nimlite/cli.nim
<https://github.com/root-11/tablite/pull/86/files#diff-7655e9fee1fd5079f3fe660f62d385a78a6d3dd0cc55c9e7f7e3819b5f924574>
(62)
- *A* _nimlite/csvparse.nim
<https://github.com/root-11/tablite/pull/86/files#diff-6bc01993fbf07b33bdad91d350966197f7f34b0f412edd8534bb5c2c8aa10bdb>
(289)
- *A* _nimlite/encfile.nim
<https://github.com/root-11/tablite/pull/86/files#diff-6008dcd0b49269647cc1becc4f6146b25bb717cceb2cb3c8dab035fb673928b4>
(181)
- *A* _nimlite/infertypes.nim
<https://github.com/root-11/tablite/pull/86/files#diff-a01e6a6bf35a35b69052367b55a617257841082756e3e6c85ee29d4ec0bc397c>
(412)
- *A* _nimlite/insertsort.nim
<https://github.com/root-11/tablite/pull/86/files#diff-55375e4e17fbbde5e651980878ff26a8f16be1926ba7873a84faed4075a300b5>
(9)
- *A* _nimlite/nimlite.nim
<https://github.com/root-11/tablite/pull/86/files#diff-b131ab4ae972a294c38d4736e05cac71d674289846f01f18b3545f43449d9d94>
(284)
- *A* _nimlite/nimlite.pyi
<https://github.com/root-11/tablite/pull/86/files#diff-131f68a2dc784da99944f382a0531a32c8c6ee7553d1cd715a169f4e90401674>
(5)
- *A* _nimlite/numpy.nim
<https://github.com/root-11/tablite/pull/86/files#diff-ce34e9979d27a230f704fb1944436fb6b90be30e61e7182c4b8ac1cef123bf30>
(47)
- *A* _nimlite/paging.nim
<https://github.com/root-11/tablite/pull/86/files#diff-f435c4ebfeeb93b4c7c112ed9e01a2058e8507bf4082b3adfc915bed4db7902a>
(219)
- *A* _nimlite/pickling.nim
<https://github.com/root-11/tablite/pull/86/files#diff-33078a327ab9e583d86c6b086fecfa4172b82e01c290b0ff3128c0eed2b91b5d>
(302)
- *A* _nimlite/pylayer.nim
<https://github.com/root-11/tablite/pull/86/files#diff-c5ffcfaf853a9f08e7ef46abcc8127687c01e1b649b148193812017ebcc83053>
(47)
- *A* _nimlite/ranking.nim
<https://github.com/root-11/tablite/pull/86/files#diff-fca3929cb33643ffe5969f0fd257a2728b12ebfd02f2d457d3392a1d4bf2df85>
(91)
- *A* _nimlite/table.nim
<https://github.com/root-11/tablite/pull/86/files#diff-1331ec1f6f60c51adaafadd198810572581ffdd75dca6c8fd244d318ea37477f>
(79)
- *A* _nimlite/taskargs.nim
<https://github.com/root-11/tablite/pull/86/files#diff-d8cc41948d15c198cf32982e2aa1d40f26c97f2521ed2277044d4eabf0453b12>
(98)
- *A* _nimlite/textreader.nim
<https://github.com/root-11/tablite/pull/86/files#diff-d0d0b86947146a0c6f8cc3a760b0f6e8432477248429b4c25005bd5c484e48e1>
(189)
- *A* _nimlite/utils.nim
<https://github.com/root-11/tablite/pull/86/files#diff-c40829e24f9554343f51c96ae749df6470e9bb76517fa2efb55b673eecc923d3>
(18)
- *M* requirements.txt
<https://github.com/root-11/tablite/pull/86/files#diff-4d7c51b1efe9043e44439a949dfd92e5827321b34082903477fd04876edb7552>
(3)
- *M* setup.py
<https://github.com/root-11/tablite/pull/86/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7>
(8)
- *M* tablite/__init__.py
<https://github.com/root-11/tablite/pull/86/files#diff-5aa2ebb194c0b2f942c1ffdea6af782ed883f0937eaece5b842eca192877c3d7>
(2)
- *M* tablite/base.py
<https://github.com/root-11/tablite/pull/86/files#diff-c308236ef3ad023e5000bcad7c0fbdfa4d1f2bf53e50f1d0c145d0b427428817>
(10)
- *M* tablite/config.py
<https://github.com/root-11/tablite/pull/86/files#diff-15f9ef21afe76caf04fb8ba499a50674dcfefb5f9648ef39444632b88eb3c652>
(6)
- *M* tablite/core.py
<https://github.com/root-11/tablite/pull/86/files#diff-334cf4bf25646d517ed024daa6437a3a1fb6f8ede20940b69dee8cfd40030234>
(2)
- *M* tablite/import_utils.py
<https://github.com/root-11/tablite/pull/86/files#diff-47edcef722d81c1f689012bd42701fc4bf400e90f4ea264678251dc8a1f75f1c>
(753)
- *A* tablite/nimlite.py
<https://github.com/root-11/tablite/pull/86/files#diff-93773b0d09695f5b0d6ba818b411d182890f7d0054141c6ca6652f005bf5ebd7>
(144)
- *A* tablite/nimlite.pyi
<https://github.com/root-11/tablite/pull/86/files#diff-1e9a3682b1aeffcd6b863c00579ff63464565a654d85180f0a3cdd81cf10f959>
(18)
- *M* tablite/version.py
<https://github.com/root-11/tablite/pull/86/files#diff-93ec861119b6d4a06abf9c83ff1d4295461f6a90d2c86d3e722476d97b8b54b9>
(2)
- *A* tests/data/utf16_be.csv
<https://github.com/root-11/tablite/pull/86/files#diff-ea8d48809f7ae848819b074702992a429d8fd4adacda7b98b8bb1da76879c8a7>
(0)
- *A* tests/data/utf16_le.csv
<https://github.com/root-11/tablite/pull/86/files#diff-ed7dfe160971ad6f4406663b12e6a7c07fa1641c06a4f8348fdaec956a5a2cf7>
(0)
- *M* tests/test_filereader_formats.py
<https://github.com/root-11/tablite/pull/86/files#diff-804f5747b1b653020ee2f167c0a2052bc62bd28f3e4525e71bd4e4f9ab857e92>
(35)
Patch Links:
- https://github.com/root-11/tablite/pull/86.patch
- https://github.com/root-11/tablite/pull/86.diff
—
Reply to this email directly, view it on GitHub
<#86>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA64MJTPCEE3W4QZHQAL3ILX5AE5JANCNFSM6AAAAAA5NQR7RU>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Codecov Report
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. @@ Coverage Diff @@
## master #86 +/- ##
==========================================
- Coverage 83.04% 82.35% -0.70%
==========================================
Files 24 25 +1
Lines 3704 3910 +206
==========================================
+ Hits 3076 3220 +144
- Misses 628 690 +62
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
added 2 commits
October 4, 2023 14:24
Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines * info Test pipelines Test pipelines Test pipelines *hmm Test pipelines * huh * WINDOWSNAKJSHBFUOIASHGFOI:AHSGOIHJASGAGHAOIUHGPAIJGSN * why is windows like this * what? ugh what Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition * what what the i'm stupid Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition Test pipelines, I hate windows edition + inlining * update yml Update ranking.nim * fallback to py impl just in case * cleanup * fix buggy nim args test derp * use cli for win * update * fixes * fix pipeline * aaaa test aa *us * test hmm hmm test ? ? ? aa ? hmm hmm huh huh revert fix test * hi * fix * huh derp hmm hmm update hmm test Update python-test.yml Update python-test.yml Update python-test.yml thonk Update python-test.yml * hmm hmm d
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm likely going to merge this on Monday, you can see if you want to add something else.
I will also rebase so that the test pipeline commits don't spam the commit history.
Known issues: I can't get it to work within Windows pipelines so I'll need help from someone with windows to get python working with nim.
I will also add a fallback to use the old python implementation as a fallback if it throws within nim until we're confident with maintaining nim only implementation.