Add an option to treat empty input as null input #1628
Comments
|
Just encountered this with an empty file. If file is empty, filter is not processed at all: Is there any workaround? |
|
bump |
|
@e1senh0rn asked:
What is the alternative behavior you have in mind? Would the -s option be useful for you? Or are you asking for new file-handing functions? Please be specific. |
|
@pkoppstein maybe the report wasn't the most accurate but it seems to me that it still refers to the original (and quite detailed) report here. |
Exactly this, jq is often used to parse output from APIs and if the API returns no output (e.g. curl returned a 403 with no body due to missing token, or 503 due to service error) than jq proceeds happily along and we're none the wiser. I think by far and wide, the user's expected behavior for |
|
Just here to say this bit me as well, specifically when using |
|
Let me rephrase the original issue; maybe it will make it more clear. I have the following json input: {
"foo": true
}and I want to use I read docs and come up with the following code (for those unfamiliar with shell, # valid input, foo is true
$ echo '{ "foo": true }' | jq -e '.foo == true' >/dev/null || echo "fail"
# valid input, foo is false
$ echo '{ "foo": false }' | jq -e '.foo == true' >/dev/null || echo "fail"
fail
# invalid input
echo '{ "foo": hoot }' | jq -e '.foo == true' >/dev/null || echo "fail"
parse error: Invalid numeric literal at line 1, column 14
failBut it does not work for the case of no input: $ echo | jq -e '.foo == true' >/dev/null || echo "fail"Obviously, foo is not true, it's not even there, but my code do not print "fail". 👎 It seems that if |
Commits a2ec1d4 and 247d465 added a few checks using jq in the following test cases: * ctr lifecycle * ctr execsync should not overwrite initial spec args * privileged ctr -- check for rw mounts Alas, those checks do not work (and never worked); jq always succeeds. This happened because 1. in `run cmd1 ... | cmd2` the part starting with the pipe character is not part of `run` statement; 2. `run` eats `cmd1 ...` output (into `$output` variable); so `cmd2` is provided with empty input. Now, 3. `jq` with empty input does not run any filters and thus succeeds (even with `-e`, see [1]). The fix is to add a separate check that the output is not empty. While at it, remove `run` where it's not needed from the other places in those three tests we fix. [1] jqlang/jq#1628 Signed-off-by: Kir Kolyshkin <[email protected]>
Commits a2ec1d4 and 247d465 added a few checks using jq in the following test cases: * ctr lifecycle * ctr execsync should not overwrite initial spec args * privileged ctr -- check for rw mounts Alas, those checks do not work (and never worked); jq always succeeds. This happened because 1. in `run cmd1 ... | cmd2` the part starting with the pipe character is not part of `run` statement; 2. `run` eats `cmd1 ...` output (into `$output` variable); so `cmd2` is provided with empty input. Now, 3. `jq` with empty input does not run any filters and thus succeeds (even with `-e`, see [1]). The fix is to add a separate check that the output is not empty. While at it, remove `run` where it's not needed from the other places in those three tests we fix. [1] jqlang/jq#1628 Signed-off-by: Kir Kolyshkin <[email protected]>
Commits a2ec1d4 and 247d465 added a few checks using jq in the following test cases: * ctr lifecycle * ctr execsync should not overwrite initial spec args * privileged ctr -- check for rw mounts Alas, those checks do not work (and never worked); jq always succeeds. This happened because 1. in `run cmd1 ... | cmd2` the part starting with the pipe character is not part of `run` statement; 2. `run` eats `cmd1 ...` output (into `$output` variable); so `cmd2` is provided with empty input. Now, 3. `jq` with empty input does not run any filters and thus succeeds (even with `-e`, see [1]). The fix is to add a separate check that the output is not empty. While at it, remove `run` where it's not needed from the other places in those three tests we fix. [1] jqlang/jq#1628 Signed-off-by: Kir Kolyshkin <[email protected]>
|
I just came across the same problem. Trying to parse the results a curl command, but when the server goes down, my curl command returns an empty string. In my jq instructions I'm explicitly trying to handle the bad state by always returning false, but when an empty string is passed through my error case isn't being run at all. I'd love to see an option that forces the rules to be run with an empty string instead of bypassing them altogether. |
|
Maybe you can do something like this: $ echo -n '{"status": "GREEN"}' | jq -esRr 'if . == "" then null else fromjson end | has("status") and .status != "RED" // false' ; echo $?
true
0
$ echo -n '{"status": "RED"}' | jq -esRr 'if . == "" then null else fromjson end | has("status") and .status != "RED" // false' ; echo $?
false
1
$ echo -n '' | jq -esRr 'if . == "" then null else fromjson end | has("status") and .status != "RED" // false' ; echo $?
false
1 |
|
That works perfectly, thank you! I didn't realize using the -sR options will cause the empty string to be passed through and then run the rules, since the rules aren't run in the other cases. So I think those two options along with the first rule you have solves this perfectly. |
|
👍 Yeap without -sR jq will run the filter on each input JSON it reads, ex: $ echo '' | jq .
$ echo '123' | jq .
123
$ echo '123 123' | jq .
123
123Which i think make sense but might be a bit surprising |
|
you can also just use: jq -n 'try input catch null, inputs | . # your code'or, if you want to be better about dealing with parse errors for the first input: jq -n 'try input catch if . != "break" then error else null end, inputs | . # your code' |
|
Yeap maybe a bit clearer than use raw slurp :) Can also do: jq -n 'inputs // null | ...' |
RE: @wader Nope, you can't do that.
$ jq -n 'true, {}, null, 0, false, 10' | jq -n 'try input catch if . != "break" then error else null end, inputs'
true
{}
null
0
false
10
$ jq -n 'true, {}, null, 0, false, 10' | jq -n 'inputs | . // null'
true
{}
null
0
null
10
$ jq -n 'true, {}, null, 0, false, 10' | jq -n 'inputs // null'
true
{}
0
10 |
|
Ah yeah good catch, i've been bitten by that before. Yes also wish jq had a something like the What about: 😄 |
|
Two more: jq -n 'reduce inputs as $i (null; $i) | ...' # null or last input
jq -n 'first(inputs, null) | ...'Ok time to do something more useful maybe :) |
|
RE: @wader
I am guessing you forgot a jq -n '[ inputs ] | .[0], .[1:][]'If the first thing you do is jq -s '.[0], .[1:][]'Yep, that's nice and short :D I am not a big fan of it since I don't like unnecessary slurps which make I think the ideal solution is: jq -n 'try input catch if . != "break" then error else null end, inputs | . # your code'run |
Ah yes i was thinking about the case when you only want the first input or null.
Oh forgot about slurp, so i guess if you only care about first value it can be: :)
Yes also like when jq uses generators over arrays to make things like that possible. |
Here is a neat tool that I made that makes heavy usage of It "partitions" arrays read from $ ./partitioner.jq 2 <<< '[1,2,3] [4,5,6,7,8]'
[1,2]
[3,4]
[5,6]
[7,8]
$ ./partitioner.jq 5 <<< '[1,2,3] [4,5,6,7,8]'
[1,2,3,4,5]
[6,7,8] |
Idea is to use the output of something later on in a filter pipeline? would a binding work instead? or maybe this would be more similar to coroutines #1342?
Nice! will have a look. For fq i added a |
|
In this post I would like to emphasize that to "branch" on whether The simplest, efficient, general-purpose way to distinguish between That is, you would replace "empty" by the program that you want to handle the case of an empty input Enjoy! |
|
Re: partitioner Note that jq has an (undocumented but internally used) builtin, |
|
RE: @pkoppstein
that doesn't handle parse errors for the first input correctly, but i don't think I understand how that is more simple/efficent/general-purpuse than the solution I mentioned: jq -n 'try input catch if . != "break" then error else "empty" end, inputs'
I know $ # i am typing [1,2,3,4,5] and [2] and then pressing ^D
$ jq -n 'def repartition(s; $n): [s] | add | _nwise($n); repartition(inputs; 2)'
[1,2,3,4,5]
[2]
^D
[1,2]
[3,4]
[5,2]
$ ./partitioner.jq 2
[1,2,3,4,5]
[1,2]
[3,4]
[2]
[5,2]
^DRE: @wader
This is probably getting a little OT, but the idea is to be able to easily read an arbitrary number of values at each iteration without having to use example: split an array into multiple arrays at every $ # with input/0
$ jq -n '[1,2,3,null,4,6,false,5,null,12,3,4,5,7,7,6][]' | jq -cn 'try repeat(1 | [ while(. != null; ([ input ]? // error)[]) ]) catch . | arrays[1:]'
[1,2,3]
[4,6,false,5]
[12,3,4,5,7,7,6]
$ # with foreach
$ # you lose values at the end if the array is not null terminated:
$ jq -cn '[1,2,3,null,4,6,false,5,null,12,3,4,5,7,7,6] | foreach .[] as $v ([ false, [] ]; if .[0] then [false,[]] else . end | if $v != null then .[1] += [ $v ] else .[0] = true end; select(.[0])[1] | arrays)'
[1,2,3]
[4,6,false,5]
$ jq -cn '[1,2,3,null,4,6,false,5,null,12,3,4,5,7,7,6,null] | foreach .[] as $v ([ false, [] ]; if .[0] then [false,[]] else . end | if $v != null then .[1] += [ $v ] else .[0] = true end; select(.[0])[1] | arrays)'
[1,2,3]
[4,6,false,5]
[12,3,4,5,7,7,6]
$ # with reduce; can only output at the end
$ jq -cn '[1,2,3,null,4,6,false,5,null,12,3,4,5,7,7,6] | reduce .[] as $v ([[]]; if $v == null then . + [ [] ] else .[-1] += [ $v ] end) | .[]'
[1,2,3]
[4,6,false,5]
[12,3,4,5,7,7,6]This is a simple example, but, when you need to take an arbitrary number of values at each iteration from an array or stream (something that you often want to do when reconstructing a value from CONS:
|
|
@emanuele6 - Please note that my two most recent posts above The first of these two posts ( It's also a case of apples and oranges with respect to your By the way, for anyone interested in a simple jq program to Example: repartition([1,2],[3,4,5],[6,7]; 2) |
|
RE: @pkoppstein
I was just trying to figure out how it was different from mine; it looked like just mine, but with the extra step of using
If you want to branch on the case in which there are no inputs with mine, you just have to write code where I wrote $ printf '\n' | jq -n 'try input catch if . != "break" then error else "empty" end, inputs'
"empty"
$ printf '%s\n' '"hello"' '"hi"' | jq -n 'try input catch if . != "break" then error else "empty" end, inputs'
"hello"
"hi"
$ printf '%s\n' '"hello"' 'hi' | jq -n 'try input catch if . != "break" then error else "empty" end, inputs'
"hello"
jq: error (at <stdin>:2): Invalid numeric literal at line 3, column 0
$ printf '%s\n' 'hello' '"hi"' | jq -n 'try input catch if . != "break" then error else "empty" end, inputs'
jq: error (at <stdin>:1): Invalid numeric literal at line 2, column 0
$
$ printf '\n' | jq -n '[try input catch infinite] | .[0] | if isinfinite then "empty" else ., inputs end'
"empty"
$ printf '%s\n' '"hello"' '"hi"' | jq -n '[try input catch infinite] | .[0] | if isinfinite then "empty" else ., inputs end'
"hello"
"hi"
$ printf '%s\n' '"hello"' 'hi' | jq -n '[try input catch infinite] | .[0] | if isinfinite then "empty" else ., inputs end'
"hello"
jq: error (at <stdin>:2): Invalid numeric literal at line 3, column 0
$ printf '%s\n' 'hello' '"hi"' | jq -n '[try input catch infinite] | .[0] | if isinfinite then "empty" else ., inputs end'
"empty"
That could be a possible solution, but it should be allowed to emit more than one value per iteration otherwise it will output arrays a lot after they were input if it gets arrays larger than for example: $ # i am entering [1,2,3,4,5,6,7], ["a","b","c","d","e"] and ^D
$ jq -cn '# s is assumed to be a stream of arrays
def repartition(s; $n):
foreach (s,null) as $a (null; # {emit, buffer}
if $a == null then {emit: .buffer}
else .buffer += $a
| if (.buffer|length) >= $n
then {emit: .buffer[:$n], buffer: .buffer[$n:]}
else .emit = null
end
end;
select(.emit|length>0).emit );
repartition(inputs; 3)'
[1,2,3,4,5,6,7]
[1,2,3]
["a","b","c","d","e"]
[4,5,6]
^D
[7,"a","b","c","d","e"]possible fix: # s is assumed to be a stream of arrays
def repartition(s; $n):
foreach (s,null) as $a ({emit: []};
if $a == null then {emit: [ .buffer // empty ]}
else .buffer += $a
| if (.buffer|length) < $n then .emit = []
else [ .buffer | _nwise($n) ]
| if (.[-1]|length) == $n
then {emit: .}
else {emit: .[:-1], buffer: .[-1]}
end
end
end;
.emit[]); |
|
@emanuele6 - To see the difference, suppose we want to branch on whether the input stream is empty. If jq had a side-effect-free version of isempty/1, we would write something like: Using the 'infinite' template, we have only to write: or, taking into account your concern: jq -n '[try input catch if . == "break" then infinite else error end] | .[0] | if isinfinite then X else ., inputs | E end' Using your approach, we would have: So the difference is now obvious: with your program, an empty input stream results in I've fixed the incremental version of |
|
Oh, right. I didn't think of that for some reason! Thank you. |
and others
Currently, if you provide an empty input,
jqappears to not run your filter program at all, which can be very puzzling (see #1497 and #1142, for example).There are times when it would be more useful to treat the empty input as a
nullvalue, so that your filter program actually gets a chance to run and you can therefore actually do something with the input (or lack thereof), react to the lack of input, and actually produce an output anyway — which currently isn't possible at all when the input is "empty".Being able to produce an output value even when the input is empty means that you could even influence the exit code when using
--exit-status(see #1497 (comment)) — also something that you otherwise couldn't do (currently always exits with 0 but is supposed to exit with 4 (#1497)).I propose adding a new
--empty-input-as-nulloption to complement the familiar existing input options like--slurpand--null-input/-n. In fact, it would work identically to--null-inputwhen the input was empty.jqusually treats its input a stream of 0 or more JSON values (separated by whitespace) ... which is great when you do in fact have multiple values coming in. But oftentimes you'll have an input script that generates a single, well-defined JSON value, and it would be more useful and intuitive to treatjq's input as a single value as well in order to process that script's single output value.This option basically gives you that: It allows users to treat the input as if it were a single JSON value, falling back to
nullif needed instead of skipping running your filter entirely and not giving you a chance to generate an output value.Examples:
This would be especially useful in combination with
--exit-status.Could use with
//operator to provide your own fallback behavior/value if you want something other thannullwhen there's empty input:The text was updated successfully, but these errors were encountered: