Raise model index limit (#40642)
* Use expression fix on both pk-ref and value-ref
The gist: inside the query expressions are referred to as [:expression
"foo"]. But as a nested query, it's just another field, and the fact
that it was an expression is irrelevant and should not leak from that
stage. So "downstream" should just be [:field "foo" base-type].
Oddly, if you just fix one field or the other, the query would be
"valid", would log an error, but no valid values would be returned.
It seems like the invalid expression reference was just dropped so we
only selected a single column and those were removed (we run `(filter
valid-tuples?)` as we only want valid key/value pairs).
So let's add a `(> (count values) 0)` which was failing previously.
* Fix indexing error
When updating the model index, it diffs the existing values vs the new
values and deletes appropriate and adds the new values. But it was using
the wrong column when updating the model index values. It used `pk_ref`
but that table uses `model_pk`. This led to the following errors on
stats:
```sql
select pk_ref, value_ref, state, error from model_index where state = 'error'
pk_ref,value_ref,state,error
"[""field"",563028,null]","[""field"",563018,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
"[""field"",534030,null]","[""field"",534037,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
"[""field"",545945,null]","[""field"",545948,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
```
Now those errors will be fixed and the model indexes will correctly
track the values
* Bump indexing threshold to 25,000
Had been limited to 5,000 rows of model indexing. This was a made up
number as we went into this endeavor. But it's proving to be far too
small. Bumping to 25,000
* Parititon deletions and inserts
With a higher limit need to ensure that we don't send too large of a
query.
```clojure
model-index=> (t2/insert! ModelIndexValue
(map (fn [id]
{:name (str (gensym "value"))
:model_pk id
:model_index_id 1})
(range 25000)))
Execution error (PSQLException) at org.postgresql.jdbc.PgPreparedStatement/<init> (PgPreparedStatement.java:105).
PreparedStatement can have at most 65,535 parameters. Please consider using arrays, or splitting the query in several ones, or using COPY. Given query has 75,000 parameters
```
* move comment and fix test failure explanation
comment should be next to the number? check. and also need to use
`mc/explain` which returns info rather than `mc/validate` which returns
a bool:
```clojure
;; bad
model-index-test=> (let [values [[1 "foo"] [2 "foo"] ["foo" "foo"]]]
(-> (mc/validate [:sequential [:tuple number? string?]] values)
(me/humanize)))
nil
;; good
model-index-test=> (let [values [[1 "foo"] [2 "foo"] ["foo" "foo"]]]
(-> (mc/explain [:sequential [:tuple number? string?]] values)
(me/humanize)))
[nil nil [["should be a number"]]]
```
Please register or sign in to comment