Funnelback 15.24.0

Release notes for Funnelback 15.24.0

Released : 23 October 2019

Supported until: 23 October 2024 (LTS) [Extended]

Improvements

In General

• Passwords stored in global.cfg and collection.cfg (e.g. passwords to be used when connecting to other systems) are now stored in an encrypted form when they are added or edited.
  • encryption.keyset-handle-provider-class can be used to customise how the encryption keys are stored or retrieved.
• Push collections now provide APIs for restoration to snapshots, including support for fetching the snapshot to be restored automatically from a remote server.
• Improved include_patterns presentation in configuration interface to better handle large sets of patterns.
• The default search results template now uses the Freemarker recommended HTML escaping mechanism rather than a legacy one.
• The implementation of search sessions and cart functionality within the default search template is now simpler and has fewer library dependencies. The upgrading to session history plugin guide can be used for upgrading existing templates to make use of the simpler variants.
• The configuration settings for contextual navigation, quicklinks and form interaction are now within the general collection configuration, allowing them to be set via the configuration interface or via the associated REST APIs.
• Improved the performance of accessibility auditor on collections covering many domains.
• /s/all-results.csv can now produce a custom csv file name by specifying the 'fileName' url parameter.
• Improved the interaction with web resources directories containing unpublished changes.
• Category headings can now be disabled in the list view on a per-entity basis within the knowledge graph widget.
• Search results data model now includes more useful debugging information related to running internal queries against search indexes.
• Improved support for faceted navigation with queries that contain bigrams (such as CJKT characters).
• Improved the task-picker such that it can load dependencies from custom 'jar' files located '$SEARCH_HOME/lib/java/task-picker/'.
• Improved query performance when many curator rules are defined for any profile under a collection.
• Improved 'build_autoc' performance for profiles reducing update times.
• Reduced the time taken by the update step ContentAuditorSummary.
• Improved support for running faceted navigation on extra searches by adding a getEffectiveExtraSearchName() method to the search transaction object which can be considered when modifying the transaction in a hook script.
• Web crawling logs now include timestamps and additional frontier debugging information.

Knowledge Graph Specific

• Knowledge graph now models metadata with multiple values in a list form, allowing for correct presentation of values containing separator characters.
• Knowledge graph configuration now allows a default template to be configured for entity types which do not have a dedicated template.
• When sorted by date, knowledge graph entities are now grouped under useful headings showing the relative date.
• Knowledge graph entities can now define a fallback thumbnail for use if some entities do not define their own thumbnail image.
• Knowledge graph entities can now be created for items with no logical URL to send a user to, and are presented without a view link in this case.
• Improved alignment of entity images and data within knowledge graph widget.

Upgrades to Funnelback components

• The default version of Moment.js used in the default template and knowledge graph widget preview has been upgraded to 2.24.0.
• Upgraded Freemarker library to 2.3.29 (from 2.3.27) which provides some new template syntax - See https://freemarker.apache.org/docs/versions_2_3_29.html.
• The Maxmind database used for queries and search analytics has been upgraded from version 1 to version 2.

Bug fixes

• Fixed an issue where the crawler would follow <meta http-equiv="refresh"> redirects that appeared within html comments. Redirects inside comments are now ignored.
• The original target page is now loaded if the user is required to log-in before accessing an admin-ui page rather than the home page.
• Fixed handling of curator rules with invalid null values.
• Knowledge graph widget now returns to the top when paging through results.
• Improved handling of knowledge graph entities with no valid node_id values.
• Restored the remove-headers collection config settings ability to remove security related headers such as X-Frame-Options.
• Fixed accessibility auditor's handling of acknowledgement and collection IDs which produced log errors and prevented 'last seen date' data being recorded.
• The configuration UI now presents very-long profile lists in a scrollable form rather than extending off screen.
• The default search template now retains the profile URL parameter when a scope removal link is followed.
• Fixed the lack of redirect information in push collection snapshots.
• Fixed handling of process-id files for Funnelback services to be compatible with recent changes to systemd.
• Fixed an issue where relationships labels in the knowledge graph widget were created incorrectly.
• Fixed an issue where sorting of related entities was not applied based on knowledge graph template configuration.
• Fixed an issue where a double request to search endpoint in the knowledge graph widget was being sent on pressing enter key when search input is focused.
• Fixed an issue where the saving of profile and server level configuration parameters were not being executed due to incorrect backup file creation.
• Tuning now uses the Perl defined in exectuables.cfg rather than any other Perl on the path.
• Prevented XSS AngularJS sandbox bypassing injection in Freemarker templates by inserting zero-width whitespace between consecutive open-curly-brackets though the default search result template no longer uses AngularJS).
• Fixed a bug in which spaces would be removed from query biased summaries which came from 'cdata' sections of XML.
• Empty XML elements mapped as the document URL are now ignored.
• Fixed a memory leak within the jetty web server by disabling the conscrypt SSL provider by default.
• Fixed errors in the sorting of faceted navigation values, which could cause an HTTP 500 error code.
• Fixed an issue where the crawler would follow <meta http-equiv="refresh"> redirects that appeared within html comments.

Important changes

• Funnelback's recommended memory requirements have been increased to reflect higher memory usage seen in practice due to new features introduced over the last few releases.
• Some license keys generated in the past will now trigger a warning saying "This collection is configured with a license in a deprecated form that will not be supported in a future version of Funnelback. Please use the license portal, or contact Funnelback, to request a new license key". To avoid issues in a future upgrade please ensure new licenses are applied to any affected collections.
• The sec.profile.manage is now used to determine whether a user can create a service rather than the broader sec.administer.system permission.
• Editing result templates in the admin UI now requires only sec.template permission.
• Editing web resources in the admin UI now requires only sec.web-resources permission.

Changes to configurations

• contextual_navigation.cfg no longer exists. Contextual navigation options are now configured directly from collection.cfg, see here for the new key names. In short, a prefix of contextual-navigation. has been added to contextual navigation keys. Additionally, the following keys have been renamed:

  • contextual_navigation_enabled -> contextual-navigation.enabled
  • type_max_topics -> contextual-navigation.type.max_clusters
  • topic_max_topics -> contextual-navigation.topic.max_clusters
  • site_max_topics -> contextual-navigation.site.max_clusters

• quicklinks.cfg no longer exists. Quick links options are now configured directly from collection.cfg, the key names and values have been retained.

• form_interaction.cfg no longer exists. Form interaction options are now configured directly from collection.cfg, see here for the new key names. The following keys have been removed:

  • crawler.form_interaction_file
  • crawler.form_interaction_in_crawl

  The following keys have been added:

  • crawler.form_interaction.pre_crawl._groupId_.url
  • crawler.form_interaction.pre_crawl._groupId_.form_number
  • crawler.form_interaction.pre_crawl._groupId_.encrypted._param_
  • crawler.form_interaction.pre_crawl._groupId_.cleartext._param_
  • crawler.form_interaction.in_crawl._groupId_.url_pattern
  • crawler.form_interaction.in_crawl._groupId_.encrypted._param_
  • crawler.form_interaction.in_crawl._groupId_.cleartext._param_

Upgrade notes

• To upgrade Moment.js to 2.24.0, update the path to resource in FTL forms from ${GlobalResourcesPrefix}thirdparty/momentjs-2.22.2/moment.min.js to ${GlobalResourcesPrefix}thirdparty/momentjs-2.24/moment.min.js. Note that previous versions of Moment.js will be removed from Funnelback in a future release.

• The upgrade process for contextual_navigation.cfg is automatic. The settings configured in contextual_navigation.cfg are moved into collection.cfg when an upgrade is run. Additionally, the relevant file permissions will be added to the <user>.ini file - assuming the user had access to collection.cfg keys and contextual_navigation.cfg.

• The upgrade process for quicklinks.cfg is automatic. Everything from quicklinks.cfg are moved into collection.cfg and preserved when an upgrade is run. Additionally, the relevant file permissions will be added to the <user>.ini file - assuming the user had access to collection.cfg keys and quicklinks.cfg.

• The upgrade process for form_interaction.cfg is automatic. The settings configured in form_interaction.cfg are moved into collection.cfg when an upgrade is run. Additionally, the relevant file permissions will be added to the <user>.ini file - assuming the user had access to collection.cfg keys and form_interaction.cfg. Please note that if there are any custom scripts which generate form_interaction.cfg files automatically, these may need to be updated to use the new implementation.

• As maxmind has been upgraded, the Location class within the search transaction at transaction.question.location has been modified. The areaCode, dmaCode and region fields have been removed. The longitude and latitude fields are now of type Double. The Location class has had the following fields added: countryGeoNameId and subdivisions. The subdivisions field is a list that generally holds the state or province that the IP address belongs to.