forked from arkime/arkime
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathCHANGELOG
More file actions
2891 lines (2736 loc) · 134 KB
/
CHANGELOG
File metadata and controls
2891 lines (2736 loc) · 134 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
NOTICE: Please see https://arkime.com/faq#upgrading-arkime for upgrading info
OpenSearch Versions:
* Arkime >= 5.7.0 supports 1.x, 2.x or 3.x
* Arkime >= 3.0.0 supports 1.x or 2.x
Elasticsearch Versions:
* Arkime >= 5.7.0 supports ES >= 7.10.0, 8.x, or 9.x
* Arkime >= 4.0.0 supports ES >= 7.10.0 or 8.x, not 9.x or later
* Arkime >= 3.0.0 supports ES >= 7.10.0, not 8.x or later
* Moloch >= 2.7.0 supports ES >= 7.4.0, not 8.x or later
* Moloch >= 2.2.0 supports ES >= 6.8.0 or >= 7.1.0, not 8.x or later
* Moloch >= 2.0.0 supports ES >= 6.7.0 or >= 7.1.0, not 8.x or later
* Moloch >= 1.5.0 supports ES >= 5.5.0, 6.x, not 7.x or later
* Moloch >= 1.0.0 supports ES >= 5.5.0, 6.x (not prod tested, only for new installs), not 7.x or later
* Moloch >= 0.50.0 supports ES >= 5.5.0, not 6.x or later
* Moloch >= 0.18.1 supports ES 2.4.x, >= 5.3.1 not 6.x or later
Node Versions:
* Arkime >= 6.0.0 requires NodeJS >= 22.15.0 or < 23
* Arkime >= 5.4.0 requires NodeJS >= 20.0.0 or < 21
* Arkime >= 5.0.2 requires NodeJS >= 18.15.0 or < 21
* Arkime >= 5.0.0 requires NodeJS >= 18.0.0 and < 19
* Arkime >= 4.2.0 requires NodeJS >= 16.0.0 and < 19
* Arkime >= 3.4.0 requires NodeJS >= 16.0.0 and < 17
* Arkime >= 3.0.0 requires NodeJS 14.x
* Moloch >= 2.4.0 requires NodeJS 12.x
* Moloch >= 2.0.0 requires NodeJS 10.x
* Moloch >= 1.6.0 requires NodeJS 8.x, 8.12 or later
* Moloch >= 1.0.0 requires NodeJS 8.x
* Moloch >= 0.20.0 requires NodeJS 6.x
* Moloch >= 0.18.1 requires NodeJS 4.x
NOTICE: Restart wiseService before capture when upgrading
NOTICE: Cross-cluster Shortcuts require you to not restart all your viewers at once after upgrading
NOTICE: Create a parliament config file before upgrading (see https://arkime.com/settings#parliament and https://arkime.com/faq#how_do_i_upgrade_to_arkime_5)
6.1.0 2026/04/xx
## BREAKING
- #3782 Previous redis user databases will be ignored
- #3786 Dedup packets including VLAN/VNI by default, set dedupVlanVni=false to disable
## All
- #3782 sqlite support added for user database and cont3xt/parliament databases
- #3782 many fixes for lmdb and redis database implementation
- #3782 now regression test sqlite/lmdb/redis databases during builds
## Capture
- #3786 add VLAN/VNI deduplication support (thanks @waynieack)
- #3790 fix VLAN parsing for Type III ERSPAN (thanks @waynieack)
## ES Proxy
- #3789 Add AWS SigV4 signing support for managed OpenSearch
## Parliament
- #3788 Fix abort when parliament file not set
## Viewer
- #3781 Most server errors are now localized
6.0.1 2026/03/09
## All
- #3768 https://[::1] automatically sets insecure now
## Capture
- #3760 Fix scheme mode not ignoring empty bpf setting
- #3762 Fix auto loading of .lua parsers not always working
- #3774 Fix GRE Enhanced v1 (PPTP) header parsing and added proper PPP framing support
- #3774 Add simple OpenVPN classifier
- #3774 Add simple Omron-FINS classifier
- #3774 Add ENIP parser
- #3774 Don't be so strict with BACnet parsing
- #3778 add simple rdpudp classifier
- #3778 add simple gtp control classifier
## Cont3xt
- #3756 Fix overview raw editing not working well
## Viewer
- #3766 Fix ips starting with :: being treated as ports
6.0.0 2026/03/02
## All
- #3747 New /api/appversion API
## ESProxy
- #3750 - fix httpsAgent race condition with client certificates
## Viewer
- #3742 ArkimeTables support i18n
- #3743 Consistent expression parser error messages
6.0.0-rc4 2026/02/23
## Release
- #3718 Build for Ubuntu 26.04
- #3726 docker.sh supports --wait-for-db option
## Viewer
- #3728 support expression autocomplete more places
## Capture
- #3724 fix ja4plus plugin to match rust implementation for edge cases
- #3731 fix crash on quit when freeing http zstrm data structures
- #3731 fix dedup increase message having incorrect values
- #3731 performance improvements with dedup and arkime_memcasestr
- #3739 disablePython defaults to true now
6.0.0-rc3 2026/02/17
## Capture
- #3699 writer-s3 always uses 0xffff for snapLen now
- #3699 writer-s3 fix gzip memory leak
- #3702 support redis:// for config
- #3706 Don't close stdin after using "-" for filename
- #3706 Cert UTCTime/GneralizedTime offset parsing fixes
- #3706 Fix rules _dropBySession not working consistently
- #3709 Fix scheme mode only queueing up to two files for later
- #3710 Fix SCTP chunk alignment, add maxSctpOutOfOrderPackets setting and check
- #3711 Fix SCTP databytes
- #3711 Fix SCTP protoid should be 32 bits
6.0.0-rc2 2026/02/10
## BREAKING
- #3677 arkime_packet_log now treats logEveryXPackets as a minimum rather
than an exact interval, and only prints TCP info
## Contrib
- #3637 increased max tzsp-forwarder packet to 64000
- #3674 added new netflow2arkime.pl script
## Capture
- #3630 tds7 protocol support
- #3637 Initial bacnet protocol support
- #3638 NTP protocol improvements
- #3640 Initial isakmp protocol support
- #3642 Initial tftp protocol support
- #3643 Improved rdp parser
- #3644 Improved snmp parser
- #3645 Improved mqtt parser
- #3651 Added basic sip parser
- #3652 Added basic stun parser
- #3653, #3666 Improve krb5 parser
- #3654 Added turn support to stun parser
- #3655 Handle different quic salts for draft23, draft29, v2
- #3655 More ssdp keywords
- #3656 Parse udp facebook quic
- #3657 Added classifiers for: plex-gdm, samsung-smartview, whatsapp, ubiquiti-ubnt, xid
- #3659 Added classifier for nbds and parser for nbns
- #3660 Added basic ptp parser
- #3661 Added isakmp cert decoding
- #3663 Added dcerpc parsing
- #3668 Added basic dnp3 parsing
- #3670 Added basic wireguard classifier
- #3672 Added some telcom protocols: m3ua, sccp, tcap, camel, diameter
- #3676 Added basic imap parser
- #3677 Align structures and remove unimportant atomic counts to help when
using large number of packetThreads
- #3678 Added classifier: gearman, esio; parser: pana
- #3681 Added synchrophasor parser
- #3682 Added s7comm parser
- #3686 Added websocket detection
- #3687 Added c122 parser
6.0.0-rc1 2026/01/26
## BREAKING
- #3138 settings parseSMTP & parseSMB removed, use disableParsers instead
- #3138 plugins must end with a supported extension, e.g. .so, .lua, .py
- #3138 setting luaFiles now defaults to no files
- #3212 with capture --scheme is now the default, use --libpcap for previous behaviour
- #3281 Remove Ubuntu 20.04 builds
- #3293 db.pl now requires a leading http:// or https:// in OpenSearch/Elasticsearch URLs
- #3306 WISE now requires webBasePath to be set if you use a non-default base path — set it in Arkime 5 before upgrading
- #3422 Cont3xt ThreatFox integration now requires an API key (free at https://auth.abuse.ch/)
- #3427 Capture now adds the first VLAN tag back to packets when saving to disk. This may affect existing BPF filters —
set tpacketv3OldVlan=true to disable.
- #3468 Digest/Form users who haven't changed their password since Dec 2019
will not be able to log in. A userAdmin can reset their passwords.
- #3473 dnsOutputAnswers defaults to TRUE now
- #3488 When talking to remote viewers, only viewUrl is used now — webBasePath is no longer used
- #3492 Viewer now expires PCAPs even if pcapDir is not set, defaulting to
/opt/arkime/raw. Previously, PCAPs were not expired when pcapDir was unset.
- #3552 Users now inherit the 7 extra permissions from their Roles unless
explicitly overridden
- #3583 Fixed: IPv4 sessions with identical src and dst IP addresses may have
had an incorrect community_id. Old sessions will retain the incorrect value.
- #3591 The geoLite2Country setting now looks for a City database file first by default
- #3601 The unkEthernet/unkIpProtocol plugins are removed. The saveUnknownPackets
setting now saves unknown/corrupt packets as real Arkime sessions.
## Release
- Node 22.22.0
- #3342 Container based on Debian 13 now
- Container includes geoipupdate
- docker.sh supports --ilm and --ism options
- #3502 FreeBSD builds
- #3518 easybutton defaults to --nothirdparty now
## All
- Migrated to Vue3!! (misc PRs)
- Remove Webpack tech debt (misc PRs)
- #3286 support oidc end_session endpoint and token if logoutUrl not set,
new logoutUrlMethod setting
- #3306 eslint upgraded to v9
- #3364 eslint vue files and enforce recommended rules
- #3468 remove support for old password storage
- #3476 new authJwsAlgorithm setting, defaults to RS256
- #3552 Users now inherit the 7 extra permissions from their Roles unless explicitly overridden.
## Capture
- #3138 lua plugin now autoloads *.lua scripts in parsers directory
if lua plugin is used
- #3208 vlan id is now stored in order seen
- #3268 New python support, *.py scripts in parsers directory auto loaded
use disablePython=true to disable
- #3357 Basic SCTP support
- #3375 For WISE/Rules fields that are lower/upper case, capture updates string
- #3427 Add first vlan back to packet in AFPacket mode
- #3460 DNS compress pointer chaining max increased to 10
- #3461 New DHCP Session linking
- #3473 dnsOutputAnswers defaults to TRUE now
- #3479 Per thread compression to ES should help with busy capture
- #3481 ArkimePacket free list, should help with memory fragmentation on busy capture
- #3494 Update field friendlyNames in db if they don't match capture
- #3501 Added reader-bpf
- #3517 Netmap FreeBSD support
- #3547 Fix erspan vlan truncating at 7 bits instead of 12 bits
- #3566 fix the sessions length being off by 1ms sometimes
- #3583 Fix community_id for v4 sessions with same src/dst port sorting
- #3591 The geoLite2Country setting now looks for a City database file first by default
- #3618 Fix S3 scheme prefix handling
- #3618 Fix S3 scheme not process over 1000 S3 items
- #3620 Simple DNS RRSIG/DS/NSEC parsing
- #3622 Added disableIp4Defrag setting
- #3623 Initial ES-IS protocol support
- #3624 saveUnknownPackets supports common strings
## Capture/Viewer
- #3197 new sessionsStarted and sessionsPresent in files tab
- #3210 new vlan.dot1q and vlan.dot1ad expressions
- #3308 City and Region from MMDB
- #3434 SCTP protoId
- #3463 Added dhcp.classId
- #3464 Added id for dhcpv6
- #3465 Added dhcp.requestIp
- #3566 New packetRange field to support spanning timeline display
- #3601 Save corrupt and unknown sessions as real Arkime sessions based on saveUnknownPackets
## Viewer
- #3326 BIG search expression
- #3343 Basic internationalization support
(most translations contributed by Cursor using Claude 4 Sonnet)
- #3341 Check files index mapping on start
- #3366 Sankey diagram on SPI Graph page
- #3374 Allow multiviewer to change password if usersElasticsearch is set
- #3376 multiviewer logs history for only clusters selected
- #3399 Now track ES node ids in dstats so on Shards tab we can show
which node is missing for node_left. ES should do this for us!
- #3423 Periodic Queries and Hunts can now notify on multiple notifiers
- #3439 multiviewer config now supports defaultCluster setting
- #3474 support 15 and 30 minute query time ranges
- #3488 only use viewUrl for remote URL
- #3492,#3536 default pcapDir to /opt/arkime/raw
- #3495 Speed improvements for add/removing tags and exporting CSV
- #3497 Process pcap files in blocks for speed improvements
- #3498 Optimize pcap ressembly memory usage for speed improvements
- #3522 Can set max scrolls and display current scrolls
- #3528 IP OR array queries should be more efficient now
- #3567 Hunts allow updating of fields while running
## Parliament
- #3395 Low disk space monitoring for capture and ES hosts
- #3395 Navbar ES status indicator cycles through clusters with issues w/highlighting
- #3395 Clickable issue table rows navigate to node stats
- #3395 Issue filters persist in URL parameters
- #3395 Toggle to show/hide all issues
- #3395 Display ES version in cluster tooltips
## Cont3xt
- #3405 Keyword/regex highlighting in integration and overview cards via ?highlight= parameter
or via Search bar mode selector to switch between query search and highlight pattern modes
- #3422 ThreatFox integration
- #3421 Zetalytics integration
- #3406 Domain Tools Whois Integration
- #3410 crt.sh integration
- #3407 Greynoise malicious tidbit
## Multies
- #3430 Handle when ES cluster returns 503 better
## WISE
- #3435 New wise urlapi source
## db.pl
- #3581 New db.pl show-nodes command
- #3600 The init/wipe/upgrade commands warn if using different settings
- #3603 Support repairing bad mapping with stats index
5.8.3 2025/12/02
## Release
- #3510 Node 20.19.6
## Capture
- #3489 Do a better job URL encoding the file exists check for managed OpenSearch
5.8.2 2025/10/27
## Release
- #3425 docker.sh add --add-admin option
## Capture
- #3414 Support tls client hello packets that are split
## Capture/Viewer
- #3408 ja4d6 support
## Viewer
- #3412 Files tab can paginate over 10k entries now
5.8.1 2025/10/20
## Release
- #3367 fix Configure not setting elasticsearchBasicAuth correctly
- New version of ja4d
## Capture
- #3363 Fix lua session.dstIp
- #3368 Fix rules reload crash
- #3401 Fix dead lock if using s3 writer and s3 scheme reader
## Viewer
- #3385 Fix missing pcap file header crash
- #3398 Fix leaving 10 oldest files since 5.4.0
- #3398 Log expire delete errors
5.8.0 2025/09/22
## Release
- #3287 Node 20.19.4
## All
- #3352 Abort and useful error for bad userAuthIps setting
## Capture/Viewer
- #3307 Added garland timestamp plugin
## Capture
- #3294 Improved dhcpv6 parser and ja4d display
- #3297, #3321 Fix lua memory leaks
- #3298 Fix dns encoding issues with answers
- #3299 Handle more than 255 ciphers with ja4
- #3305, #3337 added basic ip AH protocol support
- #3314 fix SQS/S3 with MinIO (thanks @mcgillowen)
## db.pl
- #3320 Fixed some Elasticsearch/Opensearch calls that still had types
- #3323, #3328 New db.pl repair implementation that can fix multiple issues
## Viewer
- #3319 Fix export csv from SPI Graph
- #3344 Fix parsing quoted arrays in wildcards
- #3358 Fixed files created with s3 capture scheme not showing in viewer
if not in top level directory
## WISE
- #3339 Elasticsearch source can have paths value shortcuts
- #3359 Elasticsearch source can have integer and object values
5.7.1 2025/07/23
## All
- #3259 Config stored in Elasticsearch/OpenSearch now respects --insecure
- #3279, #3274, #3273 NPM dependabot fixes
## Capture/Viewer
- #3258 Add new ethertype field
## Capture
- #3258 unkEthernet plugin now groups sessions by ethertype + macs
- #3276 fix wise lookups for some hashed string fields
## Cont3xt
- #3253 IPQualityScore support (thanks @RamboV)
5.7.0 2025/06/11
## BREAKING
- User defined roles with the user-role-mappings feature used to require
role- prefix but didn't work, now they require role: prefix and do work
## Release
- #3196 Fix Debian 13 dependency libyara issue
- #3205 arkime_config_interfaces.sh -n with dash fix
- #3211 Node 20.19.2
- #3231 No longer use screwdriver, only github actions, goodbye el7
- #3233 EL10 initial support
- #3244 Support make DESTDIR install
## All
- #3237 Add missing settings to addUser.js/arkime_add_user.sh
## Viewer
- #3199,#3200 Support searchable snapshots with partial- index prefix
- #3218 Elasticsearch 9 dstats fix
- #3224 Fix/Change user-role-mappings must start with role: instead of role-
## Capture
- #3209 New espSavePackets setting
- #3229 Drop packets larger than 0xffff in size
5.6.4 2025/04/29
## Viewer
- #3188 Prevent more session prototype pollution with connections
- #3188 Improved receiveSession auth & index verification
5.6.3 2025/04/22
## Release
- #3173 Initial Debian 13 support
## Capture
- #3175 Don't include dns.host.tokens in host query
- #3177 support autoGenerateId=consistent to reprocess into same sid
## Multies
- #3176 Fix issues when clusters are down (thanks @DavidCHIA-Rub)
## Viewer
- #3164 Support searchable snapshots with partial- index prefix
- #3167, #3169 Prevent more session prototype pollution
- #3170 Add error handling for unknown views
5.6.2 2025/03/27
## db.pl
- #3135 Support passwords > 55 characters (thanks @GhostNaix)
- #3143 new db.pl mv <node> <old prefix> <new prefix> to move many files at once
## Capture
- #3105 Support decrypted smtp
- #3136 Support ERSPAN Type III
- #3157 Log bulk FORBIDDEN errors
## Viewer
- #3137 Prevent session prototype pollution
- #3142 Fix session detail long arrays not displaying correctly
- #3147 Fix erspan decode issues
- #3148 Fix issuerCN not displaying in session detail
- #3151 Fix cert.serial not displaying in session detail
- #3158 Fix s3http/s scheme not caching blocks correctly
- #3159 Fix packets not showing up when using writer-s3 without compression
5.6.1 2025/02/13
## BREAKING
- Cont3xt Twilio integration requires a new token
## Release
- #3103 arkime_config_interfaces.sh supports interface envs
- #3121 Node 20.18.3
- #3115 build ja4 docker images
- #3127 docker.sh now sets ARKIME__usersElasticsearch (when not set) from
ARKIME__elasticsearch (when set)
## All
- #3093 if config file doesn't exist, don't exit with error. This is useful
with containers + envs. Capture does require the file to
exist if specified.
- #3107 ARKIME__ envs now use cont3xt,wiseService,... instead of default for
section name for those respective applications
- #3110 can now use https://usersElasticsearch in url/config and Arkime will
fill in from the env/config
- #3122 if no section used for override, use something sane
## Capture
- #3100 fix SSLv2 constants and misidentify DTLS 0 (thanks @droe)
- #3105 parse smtp data that isn't actually encrypted after STARTTLS
## db.pl
- #3101 support ARKIME__prefix, ARKIME__elasticsearchBasicAuth,
ARKIME__elasticsearchAPIKey envs
- #3124 new arkime_configs index for storing config files
## Viewer
- #3095 Show Arkime capture version in the stats UI
- #3114 Fix http sessions missing http request not showing body (thanks @bryangwj)
- #3120 Fix value actions not showing for info column fields
## WISE
- #3107, #3108 Support webBasePath
- #3110, #3111, #3127 if usersElasticsearch isn't set will use elasticsearch config
## Cont3xt
- #3118 update Twilio integration to v2 API
5.6.0 2025/01/15
## BREAKING
- Unknown config variables that start with tpacketv3 or simple will now cause an error
## Release
- #3051 arkime_config_interfaces.sh doesn't try and set up "dummy" interface
- #3081 afterinstall.sh uses prefix correctly
## All
- #3037 remove babel
- #3087 Env vars improvements and DASH, COLON, DOT, SLASH are now replaced
## Capture
- #3046 added packet-stats command
- #3052 add ARKIME_default__ support for env vars
- #3062 only refresh Arkime indices on exit
- #3063 use suricata vlan when using sessionIdTracking
- #3070 new --command option instead of having to use command-socket
- #3072 add ident protocol classifier
- #3079 check tpacketv3* and simple* config settings
- #3083 new _flipSrcDst rule action
- #3083 new tcp.synSet rule field
- #3083 rules can now use values of "${configvar}"
- #3088 fix memory leak if "<root>" is dns query
## Viewer
- #3055 fix missing session.network section error
- #3059 fix losing custom theme setting
- #3068 display all kinds of data nodes on ES Nodes tab
- #3076 Fix incorrect Overload Drops/s statistic in Capture Stats page (thanks @mcgillowen)
5.5.1 2024/11/20
## Release
- #3011 Add db.pl to docker.sh
- #3015 Node 20.18.0
- #3021 docker.sh now supports --init and installs missing iproute2 package
## All
- #3010 fix lmdb cont3xt and users DB
## Cont3xt
- #3012 add basic databricks support
- #3016 fixed cont3xt health check request every second - should be 10s
## db.pl
- #3017 New field-list, field-rm commands
## Viewer
- #3008 fixed sessions column sorting not working in some cases
## WISE
- #3012 add basic databricks support
5.5.0 2024/11/11
## Release
- #2925 Node 20.17.0
- #2956 CyberChef 10.19.2
- #2992 Now have official docker container at https://github.com/arkime/arkime/packages
## All
- #2947 new user-role-mappings section for oidc/header auth
- #2950 support authRedirectURIs list (thanks @divinehawk)
- #2954 Fix form/oidc authMode failure to start when deleting old sids failed
- #2964 Add to files tab lastPacket timestamp and start/finish processing time stamps
- #2995 Switch to arkime-iptrie
## Capture
- #2924 _closeNow rule operator
- #2929 Update ja4 for alpn edge cases
- #2940 cert.ja4x* now work with rules/wise
- #2959 New --libpcap option for libpcap offline processing vs
--scheme for new faster method
- #2969 Add back host.dns to rules
- #2991 Add initial IP TTL and TCP Seq number fields
- #2996 pcapDir defaults to /opt/arkime/raw and pluginDir defaults to /opt/arkime/plugins
## db.pl
- #2946 fix sync-files not handling multiple nodes, or dash containing nodes
correctly (thanks @dennisse)
## Multies
- #2962 fix caTrustFile not working with multies
## Viewer
- #2926 cronQueries=auto now uses the node name in the unique key
- #2935 spigraph treemap shows unique Dst/Src IPs
- #2945 add iframe 'allow' option
- #2965 fix millisecond timestamp setting not saving
- #2966 Add the ability to hide tags in the session table
5.4.0 2024/08/05
## Release
- #2885 Node 20.15.1
## All
- #2892 backoff recurring health requests if they fail
- #2898 support using env vars for many config settings (thanks @Jc2k)
## Capture
- #2866 for s3/sqs scheme support standard AWS credentials methods including
env vars, --profile ~/.aws/credentials or config, and meta data service
- #2869 scheme mode for local files support monitor mode
- #2870 log error with pcap_dispatch (thanks @vpiserchia)
- #2873 New --command-socket option to enable a unix domain control port for
controlling capture
- #2875 New --command-wait option to use when no offline files on command line
- #2877 command-socket add-dir now has options to override command line
- #2891 fix JA4 when num extensions or ciphers is > 99
- #2893 support deleting pcaps when ignoreErrors set (thanks @vpiserchia)
- #2894 support --op/--delete with scheme commands
## Cont3xt
- #2879 Added skipChildren query string parameter
- #2880 Only focus on search if no search parameter
- #2890 Date formatting in link groups
- #2903 Added Quad9 tidbits on domains and ips
- #2904 Added Email Reputation integration
## Multies
- #2865 form or oidc require usersElasticsearch to be set for multiES
## Viewer
- #2884 Improve hunt parallelization, run 2 sessions per node at once
- #2886 Expire logging improvements and multiES disable
- #2896 added protocols/tags to default info column fields
- #2899 fixed viewer not loading writer-s3 files since Arkime 5.3.0
- #2900 Support hunts on multiviewer, still need a normal viewer to run
hunts
5.3.0 2024/06/27
## Release
- #2821 CyberChef 10.19.0
## All
- #2842 requiredAuthHeaderVal can be a comma separated list
## Capture
- #2820 fix puny dns entries missing from all list sometimes
- #2832 arkime_config_interfaces.sh fixes (thanks @dennisse)
- #2833 support processing a directory from s3
- #2835 fix crash with bgp parsing and shifting time
- #2835 create pcap files with at least config snapLen
- #2835 fix files with no processed packets hanging capture
- #2856 suppport AWS SQS for notifications to process new S3 files
- #2856 fix scheme pcap processor with corrupt pcap files
- #2859 log error if maxStreams is too low
## Cont3xt
- #2823 added "Add Field" button to top of overview form
- #2829 new DNS integration card
## Multies
- #2853 don't initialize Auth subsystem
## Parliament
- #2849 fix form auth not styling correctly
- #2857 display cont3xt/wise links for non admins
## Viewer
- #2817 new maxSessionsQueried setting, default 2MM
- #2819 hide noFacet fields from being columns
- #2824 add ability to delete shards from ES Shards tab
- #2834 improve forcedExpression help
- #2840 fix downloading pcap of scheme uploaded items
- #2843 new %NODEHOST% substitution
- #2845 fix s3 download to work with LocalStack
## WISE
- #2834 fix azure ip link
5.2.0 2024/05/28
## BREAKING
- db.pl upgrade is required when upgrading from 5.1.2 or earlier
## Release
- #2779 Using https://localhost or https://127.0.0.1 no longer requires --insecure
- #2779 Configure now prompts for OpenSearch/Elasticsearch user/password
- #2783 New debian 12 package
- CyberChef 10.18.3
- #2797 Node 18.20.3
## All
- #2772 fix OIDC login crash
- #2773 Users tab saves automatically on change
## Capture
- #2748 icmp community id support
- #2766 VLAN or VNI can be used in session ids, controlled by sessionIdTracking
- #2784 Added DTLS JA4 support
- #2791 udp databytes was too large when padded
- #2796 Added DTLS JA4S support, requires new ja4plus plugin also
- #2799 support multiple EOL for JA4T and JA4TS
## Cont3xt
- #2795 support adding clickhouse integrations
## db.pl
- #2798 fix expire/rotate warnings
## Viewer
- #2741 save session table info column fields
- #2800 scrubbing pcap of compressed/encrypted packets deletes references
instead of failing
5.1.2 2024/04/23
## Release
- #2759 CyberChef 10.17.0
## Capture
- #2756 parse SMB dialect
- #2758 fix rules not always matching "0" for non array integer fields
## Viewer
- #2765 add esadmin functionality to multiviewer
5.1.1 2024/04/15
## Release
- #2752 Node 18.20.2 (EL 7, Ubuntu 18 still on 18.19.1)
## Capture
- #2732 rules support NOT string and integer fields
- #2746 fix DNS parser PUNY length checks
- #2744 fix empty pcap files hanging capture
## Viewer
- #2745 don't autocomplete values starting with a quote
5.1.0 2024/04/04
## Release
- #2667 support Node 20
- #2734 Node 18.20.1 (EL 7, Ubuntu 18 still on 18.19.1)
- #2737 CyberChef 10.15.0
## Capture/Viewer
- #2694 New DNS parser that captures all the answers, enable with
dnsOutputAnswers=true (thanks @mcgillowen)
## Capture
- #2674 Fix filelist not working in scheme mode
- #2679 cert.alt can be used in rules
- #2699 Disable reader s3 download timeout
- #2726 Fix ZSTD_decompress missing for some builds
## Cont3xt
- #2683 lock integration settings
- #2719 snap to dates
- #2730 Arkime/OpenSearch/Elasticsearch integration had insecure logic backwards
## Viewer
- #2668 fix pcap export with only default time range and no date param in url
- #2680 add default user settings to viewer config
https://arkime.com/settings#user-setting-defaults
- #2681 fix unique of numerical fields
- #2701 Make sure pcap reassembly doesn't starve viewer
- #2704 Support viewUrl having a path
- #2705 Support querying non Arkime indices, enable with
queryExtraIndices (thanks @mmguero)
- #2718 Green on black theme improvements, Elyse's fav now
- #2735 help improvements
- #2736 help improvements
5.0.1 2024/02/20
## Release
- #2631 CyberChef 10.6.0
- #2648 Build for Ubuntu 24.04
- #2655 Support rpm fips installs again
- #2558 Node 18.19.1
## Capture
- #2634 add esp packet stats (fixes #1116)
- #2638 support readTruncatedPackets on live captures
## db.pl
- #2633 noprompt outputs less warnings
- #2639 fix init not working with large number of indices
## JA4+
- Fixed memory leak
- Fixed JA4H issue with long cookies
## Parliament
- #2645 Fixed issues not being detected
- #2659 Fixed parliament crashing if userPrefix not set
## Viewer
- #2632 fix field labels not expanding fully
- #2637 fix session detail grip
- #2668 fix pcap export with only default time range and no date param in url
## WISE
- #2653 set a threatstream.indicator field
5.0.0 2024/02/06
## BREAKING
- #2297 s3Compression/simpleCompression now defaults to zstd
- #2297 s3WriteGzip removed, use s3Compression=gzip for gzip instead of new zstd default
- #2297 s3GapPacketPos defaults to TRUE
- #2297 enablePacketDedup defaults to TRUE
- #2299 #2308 authMode defaults to digest now
- #2312 removed old v1 viewer APIs
- #2349 parliament password removed, must configure common auth via the UI before upgrading or manually in the config file see [parliament](https://arkime.com/settings#parliament) and [how do I upgrade to 5](https://arkime.com/faq#how_do_i_upgrade_to_arkime_5)
- #2402 WISE/tagger must now use http.request.FIELD/http.response.FIELD when referencing header defined with headers-http-request/headers-http-response
- #2450 Centos 7 build no longers includes pfring support
- #2453 Increase simpleCompressionBlockSize default to 64000
## Release
- #2448 zstd 1.5.5, nghttp2 1.57.0, maxmind 1.7.1, yara 4.2.3
- #2443 Centos 7, Ubuntu 18, Alpine use unofficial builds of node
- #2543 node v18.19.0
- #2447 support building on alpine
- #2549 use configure prefix more places (thanks @vpiserchia)
- #2584 AL2023 & Ubuntu22.04 ARM builds
## All
- #2316 programs support same config file formats (ini/json/yaml) and retrieval (file, elasticsearch)
- #2419 json/yaml config file formats now allow arrays instead of comma/semi separated
- #2299 #2308 authMode setting added
- #2299 #2408 #2463 added authMode: basic, form, basic+form, basic+oidc, headerOnly, header+digest (same as header), header+basic
- #2387 notifiers for parliament and arkime merged conflicts mitigated by appending "Parliament" to parliament notifiers
- #2396 drop privileges is now AFTER http(s) list
- #2509 add optional login message for form auth
- #2511 new authOIDCScope setting
- #2482 new logoutUrl setting
- #2571 new scheme pcap reading
- #2618 better error message when can't use OpenSearch/Elasticsearch on startup
## Capture
- #2295 moloch converted to arkime
- #2312 override ips can now set any field
- #2312 overrideIpsFiles setting
- #2314 packetDropIpsFiles setting
- #2390 can have negative cert.validDays/cert.remainingDays (thanks @mcgillowen)
- #2390 added cert.remainingSeconds/cert.remainingSeconds (thanks @mcgillowen)
- #2390 cert.remainingDays is now based on the firstPacket of session instead of current time (thanks @mcgillowen)
- #2409 JA4 support
- #2409 JA3/JA4 support for smtp STARTTLS
- #2297 always build zstd (except arch)
- #2517 new custom-fields-remap feature
- #2186 count the number of http methods per session
- #2528 new oui.txt location, some names have changes, fixes #2347
- #2539 new tls:has_esni tag if the client hello has esni
- #2553 fix rules range matching not working always
- #2554 support fieldSet tcpflag rules
- #2575 fix startup complaint about aliases, category, and transforms
- #2576 support different dlt for pcap-over-ip
- #2592 fix sometimes not identifying quic protocol correctly
- #2600 add tls:has_ech tag (thanks @renini)
- #2614 new kafka-config section
- #2622 fix malicious quic packet crashing capture
## Cont3xt
- #2121 new bulk UI and support for bulk queries
- #2271 lots of keyboard shortcut improvements
- #2383 new array syntax for links substitution
- #2382 new OpenSearch/Elasticsearch integration (config file only)
- #2441 new csv/json file/url/redis integration (config file only)
- #2385 new viewRoles in config file per integration to control access
- #2407 transfer ownership of resources
- #2437 new csv/json data source supports
- #2441 new redis data source support
- #2507 demoMode added
- #2527 skipChildren added
- #2532 new wise integration
- #2580 add links to integration search page from card
- #2565 added punycode decoding
## db.pl
- #2588 db.pl won't try and backup indices that don't exist
- #2588 db.pl backup cont3xt indices
## ESProxy
- #2483 #2484 support field updates/deletes
## Viewer
- #2296 removed x-moloch-auth
- #2392 files/history/stats now have cluster dropdown for multiviewer
- #2402 http.request.FIELD and http.response.FIELD supported
- #2404 add editor for resources
- #2407 transfer ownership of resources
- #2482 added uploadRoles to control who can upload
- #2501 add defaultTimeRange setting
- #2521 add footerTemplate setting
- #2525 add [config setting](https://arkime.com/settings#spiViewCategoryOrder) to set spiview category order
- #2523 resize session detail field label/values
- #2552 added %URIEncodedText% for URI encoded substitution (thanks @vpiserchia)
- #2574 fix longstanding issue with backslash search and SMB
- #2601 patch cyberchef xss vuln (https://github.com/gchq/CyberChef/issues/1468)
- #2606 zstd sometimes didn't read all packets
- #2607 improved session detail display
- #2621 session detail link a link now, multi select info column items now
## Parliament
- #2377 dashboard-only mode removed, if you want users to just see the dashboard don't assign them the parliamentUser role
- #2395 configuration is now stored in opensearch/elasticsearch
- #2530 add Users page
## WISE
- #2537 new urlScrapePrefix/urlScrapeSuffix used with urlScrapeRedirect
- #2537 new jsonl format supported
- #2588 don't setup auth if --webconfig isn't used
4.6.0 2023/10/16
- release - curl 8.4.0
- release - fix viewer systemd file
- capture - fix zstd hanging capture on full buffer
- viewer - corrupt http session decoding might hang viewer
- viewer - handle uncompressing pcap errors better
- viewer - role check in UI didn't always work
- all - handle cookies encoded with bad proxy
4.5.0 2023/09/13
- release - node 16.20.2
- release - added missingok to default logrotate for arkime
- capture - dns answers were double parsed
- capture - custom-fields honors viewerOnly:true
- capture - added dns.https fields
- capture - added cert:certificate-authority tag (thanks mcgillowen )
- cont3xt - remove raw view button for link groups on the cont3xt search page
- cont3xt - Overview shortcut
- cont3xt - fixed overviews not updating on switch
- db.pl - don't allow '.' to be used for sync/add path
- viewer - fixed ipv6 session display issues when :: in ip
- viewer - http display rewritten to not depending on nodejs internals
- viewer - gpe display improvements
4.4.0 2023/08/02
- release - cyberchef 10.5.2
- release - update arkime_update_geo.sh to use different manuf location
- all - improved json verification
- all - better logging when requiredAuthHeader fails
- all - better role creation/usage validation
- all - don't allow circular role dependencies
- all - now need to be an userAdmin and *Admin to update *Admin change
settings for another user
- all - more auth debugging
- all - can now change the password of another *Admin user if you have
userAdmin and all the same *Admin
- all - hide webEnable, headerAuthEnable checkboxes for roles
- all - oidc now uses sameSite: Lax instead of sameSite: Strict for cookies
- capture - handle tcp port reuse better
- capture - fix kafka memory leak when produce fails
- cont3xt - New overview cards
- cont3xt - fix startup race condition with db init
- cont3xt - new search protocol to prepare for bulk
- parliament - fix parliament clean start not letting auth be set up
- viewer - gtp decoding
- viewer - demo mode improvements, arkimeAdmin can use normally
- viewer - fix unique endpoint not enforcing user time limit
4.3.2 2023/06/13
- release - cyberchef 10.4.0 libpcap 1.10.4
- all - config 'prefix' can be at most 50 characters
- all - new cookie generation code
- capture - handle packets better at epoch time
- cont3xt - add twilio country code tidbit
- cont3xt - add httpRealm to sample config
- cont3xt - help improvements
- cont3xt - minor UI improvements
- db.pl - set ISM deleteTime for sessions correctly
- esproxy - add tests
- parliament - fixed occasional missing token error
- viewer/wise - Field/Value actions now support all:true to show on every instance
- viewer - Fix Src/Dst mouse over for packets/bytes
- viewer - Field Actions didn't work in expanded meta
- viewer - Fix sending/receiving sessions not working
4.3.1 2023/05/08
- BREAKING - If running mixed versions of Arkime, broken cron queries error
might show on OLD version
- release - fix ubuntu22 kafka dep
- all - passwordSecret log message now has the right [section]
- capture - --tags option now works as well as --tag
- viewer - new auto cronQueries setting
- viewer - change where primary viewer info is stored to not cause constant
mapping change
- viewer - fixed ipv6 not working, now assumes zero filled with mask (if
not provided)
- viewer - code refactor into javascript classes
4.3.0 2023/04/27
- BREAKING - Only SuperAdmin can assign *Admin roles now
- release - fix kafka library linking
- release - al2023 support
- release - improve arkime_config_interfaces.sh
- release - Configure doesn't offer demo Elasticsearch on Arch
- release - reqBodyOnlyUtf8=true in sample config file
- all - support colon in OpenSearch/Elasticsearch password
- all - fix some prototype pollution
- all - improve roles enforcement
- all - New authTrustProxy setting
- capture - tcpClosingTimeout setting controls delay before saving tcp
sessions after close
- capture - default dbBulkSize to 1M, min 500K, max 15M and removed
from sample config file
- capture - s3 writer now writes multiple files based on packetThreads
- capture - s3 writer supports zstd, s3Compression setting
- capture - s3 writer compression level, s3CompressionBlockSize setting
- capture - s3 writer block size, s3CompressionBlockSize setting
- capture - s3 writer gap encoding, s3GapPacketPos setting
- capture - s3 writer when s3UseECSEnv is true use container env vars to find
the id/key/token for s3 auth
- capture - improve Gh0st parser (#2225)
- capture - new dnp3 & finger classifier
- capture - tcphealthcheck adding debugging
- capture/viewer - includes setting ignores missing files starting with -
- cont3xt - add malicious tidbit from urlscan results
- cont3xt - add malicious and brand columns to results table for urlscan
- cont3xt - link group UI improvements
- cont3xt - add createDate for whois data
- db.pl - new --ifneeded option to init/upgrade that will exit if not needed
- parliament - fix digest auth
- parliament - better auth support
- parliament - improve issue page and filters
- viewer - display errors when cronQueries isn't configured
- viewer - fix first sessions table row obscured sometimes
- viewer - disable more apis in demo mode
- viewer - allow roles forced expression without user forced expression (#2213)
- viewer - s3 now use each file's bucket to determine access style
- wise - only send csp headers in initial request for wise page
4.2.0 2023/03/01
- release - node 16.19.1, support node v18
- release - fix arch build issues
- release - EL9 build uses sha256 digest
- all - OpenSearch/Elasticsearch name cleanup
- all - cleanup nodejs dependencies
- all - refactor how authentication is done, everything now uses passportjs
- all - support oidc authentication method
- all - caTrustFile setting should work everywhere
- capture - support ERSPAN Type I and vlan for Type II
- capture - new kafka plugin for sessions
- capture - use malloc instead of GSlice
- capture - corrupt DNS alt name memory leak fixed
- capture - Added simpleFreeOutputBuffers setting
- cont3xt - raw create link groups
- cont3xt - two clicks to delete link groups or links
- cont3xt - classify domains with multiple dashes correctly
- cont3xt - added ability to copy links between link groups
- cont3xt - support intl phonenumbers
- db.pl - Initial OpenSearch ISM support
- db.pl - Better error text for cert verify failure
- esproxy - fix converting basic auth to base64
- viewer - fix field actions crash
- viewer - can now use expression http.request.FIELD or http.response.FIELD
with headers-http-request, headers-http-response defined fields
- viewer - support viewing ipv6 DLT_RAW (#1293)
- viewer - ESAdmin -> Unflood works on users cluster now also
- viewer - support running in s2s auth mode only
4.1.0 2023/01/10
- release - glib 2.72.4 cyberchef 9.55.0 flot 4.2.3 d3 7.7
- db.pl - backup/restore wasn't dealing with templates correctly
- db.pl - upgrade failed if there was no moloch_shared user
- db.pl - repair now fixes missing history/ecs templates
- db.pl - fix users-export/users-import
- cont3xt - support missing auth and userTmpl settings
- cont3xt - Hide link group when no links match filter
- cont3xt - Added landing page
- capture - allow wise field dst.ip:port
- capture - add VNI field
- capture - initial tzsp reader support
- capture - y2038 fixes
- capture - Integer ops in rules now support a leading min or max which only
sets the value if less than or greater than current value
- wise - added usersElasticsearchBasicAuth setting and lmdb cache support
- wise - add passivetotal value action if at least key is defined
- viewer - fix es node stats for different node.roles
- viewer/cont3xt - can now search roles
- viewer/cont3xt - don't show change password menu item if web auth is enabled for
user and disableUserPasswordUI is true
4.0.3 2022/11/28
- release - cyberchef 9.54.0
- release - copy systemd files instead of soft linking
- releaes - capture/viewer systemd files now After OpenSearch/Elasticsearch
- capture - on short runs, field definitions weren't getting updated
- capture - s3 writer sets s3Compress to false with s3WriteGzip true
- capture - JA3s value was sometimes incorrect
- cont3xt - fixed digest mode fetching settings from config file
- db.pl - fixed init not working with OpenSearch sometimes
- db.pl - will now count data or data_hot node roles
- viewer - fixed showing more than 10 roles
4.0.2 2022/11/01
- release - cyberchef 9.48.0
- all - better console output sanitization
- capture/viewer - Add TLS Certificate Organisational Unit field parsing (PR #2038)
- capture - use arkime_update_geo.sh in error msg
- capture - log error and exit if fields loading fails
- release - Stop Configure from destroying systemd files
4.0.1 2022/10/18
- addUser.js - remove WARNING adding first user
- addUser.js - --webauthonly now sets header auth flag
- all - better console output sanitization
- capture - offline pcap allows more outstanding packets based on maxPacketsInQueue
- db.pl - Fixed some OpenSearch compatibility
- db.pl - Fixed upgrading to 4.x with no _moloch_shared user
- viewer - Fix cert notbefore/notafter showing bad dates in sessions table
4.0.0 2022/10/11
- BREAKING - Must be 3.3.0+ to upgrade to 4.x
- BREAKING - systemd files auto installed, still need to enable
- BREAKING - Move to roles for some permission checking,
userAdmin role required to edit users
- BREAKING - the version file lives in common directory now
- BREAKING - new defaults maxFileSizeG=12, compressES=true
- BREAKING - pcap compression is turned on by default, disable with simpleCompression=none
- BREAKING - simpleGzipBlockSize renamed simpleCompressionBlockSize
- BREAKING - right-click changed to value-actions in config
- BREAKING - the userId search in history for admin nolonger adds the surrounding wildcards automatically
- BREAKING - views & notifiers are now their own indices
- release - cyberchef 9.46.5, node 16.16.0
- release - systemd files are delivered with /opt/arkime path instead of setting at install time
- release - CICD tests with OpenSearch
- all - Support ES 8 & OpenSearch
- all - check for missing users index or no users on startup
- all - update code/docs to mention OpenSearch
- addUser.js - new --roles option, --admin creates superAdmin user
- capture - New ecsEventDataset setting
- capture - save sessions not saving packets for across restarts
- capture - afpacket rewrite, improve performance & less out of order packets