Skip to content

Release notes

Funnelback 15.18.0

Released : 3 December 2018

Supported until: 3 December 2019 (Short Term Support Version)

15.18.0 - New features

  • Funnelback Knowledge Graph
    • An extension to Funnelback search that analyses data from your enterprise systems, links it together and then presents it in a browsable interface. Enables people to more easily find and contextualise organisational information.
  • Redesigned user management system including:
    • Greatly improved user interface for creating and managing users.
    • Support for roles to streamline user permission management.
    • Ability to restrict user management permissions based on a user list or username suffixes.
    • Documentation of all permission settings within Funnelback.

15.18.0 - Selected improvements and bug fixes

  • Sessions API cart.json now sets the collection to the one which provided the URL rather than setting it to the meta collection. Note the value is only changed for values in the cart added after the upgrade.
  • Upgraded Tika to version 1.19.1
  • Introduced a boolean-expression lock-key-matcher and a raw lockstring storage mode.
  • Added a security manager to prevent hook scripts calling System.exit (which would terminate the web server process).
  • Fixed MetadataScraper applyIfNoMatch logic.
  • Fixed MetadataNormaliser to work correctly with the new filter framework and under push collections.
  • Introduced the ability to return the complete summarisable text for documents in results.
  • Changes to allow many collection.cfg settings affecting query-time behaviour to be set in profile.cfg
  • Funnelback now bundles the current OpenJDK 8 build from AdoptJDK rather than Oracle's version.
  • Fixed handling of absolute paths in the crawler.form_interaction_file setting.
  • Fixed handling of some special characters in reporting blacklist for pattern analyser.
  • Added processing of wmeta weighting for phrase queries.
  • Added new query processing laxity and sort_ignoring_tiers options.
  • SAML Identity provider initiated login now redirects to the marketing dashboard.
  • Avoided the web crawler potentially loading sitemap.xml files multiple times.
  • Fixed several cases where SAML authentication did not operate as expected in some web browsers or when the funnelback_documentation collection was inaccessible.
  • Removed some no-longer-supported Facebook permissions and fields from the Facebook gatherer.
  • Added fields to Twitter documents containing reply related metadata.
  • Fixed web crawler link decoding behaviour.
  • Fixes support for sort mode '3' in query completion, allowing the 'alpha' setting to be respected.
  • Improved the performance of the directory gatherer by caching attribute syntax definitions.
  • Fixed the default form-not-found template which reflected the given form id without proper escaping.
  • Fixed the default configuration of URL previewing which could previously be used to expose local log file content.
  • Push slaves will now actively pull down merge/vacuumed generations, rather than waiting for commits to trigger this.
  • Under Browse Collection Configuration Files users will be able to list/view/edit/etc files under all profiles which exist and the which the user has access to without specific file manager rules being created.

15.18.0 - Configuration Upgrade Steps

The following changes will be automatically performed on all configurations during the upgrade process. Configurations migrated from older versions after the upgrade will need to have update-configs.pl manually run to apply these changes.

  • Users with a pre-15.18 user-info section will be upgraded to the new 15.18 format in which this section is renamed to user-details. The 15.18 format changes the interpretation of the profile setting within this section such that an empty value now denotes access to no profile rather than to all, with * granting all-access instead. This makes the behaviour consistent with collections and other similar settings.
  • Users who previously had the sec.administrator permission are granted permission to:
    • Create and delete users and roles (sec.accounts.create-users, sec.accounts.delete-users, sec.accounts.create-roles, sec.accounts.delete-roles)
    • Change user passwords (cp.change.passwd)
    • Edit and grant all roles (roles -> can-edit-roles = *, roles -> can-grant-roles = *)
    • Edit all users (user-details -> can-edit-users = *)
    • Create roles and users with any suffix (roles -> can-create-roles-with-suffix = *, user-details -> can-create-users-with-suffix = *).
  • All users are granted sec.knowledge-graph-labels, sec.knowledge-graph-relationships and sec.knowledge-graph-templates to allow customisation of knowledge graph relationships and presentation.
  • All users have a version section with the setting version=15.18 added to support any future upgrade operations.

15.18.0 - Upgrade Issues

  • The Sessions database schema has been changed, so for installations using external databases the update-session-db.groovy will need to be run (see Search sessions and history for instructions). The script will be run automatically on upgrade for installations using the default sessions DB.
  • The MetadataScraper filter has fixed its implementation of rules with applyIfNoMatch: true. Previously, setting this flag on a rule meant that it would apply on all documents (i.e. even if the selector did not match). This has now changed so that the rule only applies when the selector does not match. Any implementations relying on the old behaviour will need to be updated accordingly.
  • The following permissions will now generate a warning if present in user ini files as they have either been removed or upgraded in a previous version or they where never used cp.status, plugin.agencies.help, plugin.build.report.db, plugin.emergency.contacts, plugin.ip.records.tracker, plugin.mgt.clients, plugin.publish.config, plugin.qld.logs.archive, plugin.show.fm.rules, sec.accessibility-auditor.admin, sec.accessibility-reports, sec.content.optimiser, sec.edit.advanced, sec.edit.clients, sec.fareporter, sec.kill.doc, sec.kiosk, sec.perform.feed, sec.qldagency, sec.service.manifold, sec.service.manifoldcf, sec.stop.manual, sec.superuser, sec.synonyms, sec.view.collection, sec.wcag.

Funnelback 15.16.0

Released : 7 August 2018

Supported until: 7 August 2019 (Short Term Support Version)

15.16.0 - New features

  • Funnelback licenses are now assigned per-collection rather than per-server, allowing multiple licenses to be used on a single server.
  • Long running tasks such as collection updates and search analytics processing are now submitted to a task-queue which can be customised to delay new tasks when the Funnelback server is under heavy load.
  • Introduced support for searching Slack messages via the new Slackpush collection type.
  • Introduced dedicated collection types for Facebook, Flickr, Twitter and Youtube, removing the need to create custom collections for these types.
  • Bulk CSV import/export of best bets, allowing for offline editing in a spreadsheet.

15.16.0 - Selected improvements and bug fixes

  • Added management screens for crawler site profiles.
  • Added metadata selection dropdown options within faceted navigation configuration.
  • Added facet selection dropdown options within curator configuration.
  • Introduced 'listMetadata' in the search result data model, which provides pre-separated values for each metadata class based on the defined separator characters.
  • Added ability to access requestHeaders via the searchQuestion data model.
  • Improved performance of push API when using the multi-part endpoints.
  • Introduced daemon.max_heap_size, jetty.max_heap_size and jetty.max_metaspace_size global cfg options to persist memory adjustments between upgrades.
  • URL facets have been improved so that it works better in cases where the URL contained non indexable characters, the URL path contain repeated path names e.g. /foo/foo/foo as well as some fixes to case sensitivity.
  • Introduced crawler.send-http-basic-credentials-without-challenge setting (on by default) to match old crawler behavior of sending http basic credentials without an initial 401 challenge.
  • Jetty has been upgraded to 9.4.11.v20180605 and the multi-part parser has been changed to a RFC7578 compliant parser which is stricter than the previous multipart parser. The multi-part parser is faster which is especially useful for the push API.
  • Jetty now uses the conscrypt SSL library which results in jetty using more secure and faster SSL ciphers. Java clients to Funnelbacks APIs should switch to using conscrypt to take advantage of the faster encryption, otherwise your client will likely be slower than it was. The push API client which uses funnelback-api-client-core.jar can be upgraded to use conscrypt by getting a copy of $SEARCH_HOME/lib/java/all/funnelback-api-client-core.jar.
  • Multiple changes have been made to the Push API to improve its performance.
  • Improved support for binary file filtering (Apache Tika upgraded to 1.18).
  • Upgraded embedded version of Java runtime, which now includes the Java Cryptography Extension. Previous versions required manual installation in some SAML use cases.
  • Analytics updates now supports multiple collections updating at the same time.
  • Analytics update pre_reporting_command and post_reporting_command is now run with the collection reporting lock held, which means while they are running another analytics job will not be able to run.
  • Fixed reading of server.cpu_count global.cfg option, as some places were using the key cpu_count which would result in the default value of auto being used.
  • Removed complexity check which prevented contextual navigation running in some cases.
  • Added ability to re-apply gscopes on local collections.

15.16.0 - Configuration Upgrade Steps

The following changes will be automatically performed on all configurations during the upgrade process. Configurations migrated from older versions after the upgrade will need to have update-configs.pl manually run to apply these changes.

  • Users with access to the old cp.license.key permissions will be granted the new sec.license.view-usage, sec.license.can-edit-other-users-licenses, sec.license.install and sec.license.delete permissions.
  • Users with access to the relevant files in the file manager are now granted the following new permissions sec.spelling, sec.url-kill-list, sec.reporting-exclusion, sec.server-alias, sec.site-profile.

15.16.0 - Upgrade Issues

  • Search request IP addresses are now pseudonymised by default - See ui.modern.pseudonymise_client_ips to disable this if needed.
  • As Funnelback now supports multiple licenses per installation some APIs are no longer possible and have been removed.
    • GET /admin-api/license/v1/usage API has been removed and replaced with GET /admin-api/license/v2/document-usage-per-license, which returns usage for all licenses the user has permission to use as well as all licenses that are used in collections the user has access to. This new API, like the old, respects sec.license.view-usage.
    • GET /admin-api/license/v1/details API has been removed and replaced with GET /admin-api/manage-licenses/v1/licenses, which returns all details for all licenses the user has permission to use.
  • The default timeout for contextual navigation has been reduced from 5 seconds to 1 second. Collections relying on the old default may need to set the timeout value.
  • Added support for Facebook Graph API version 3.1 by upgrading the RestFB library from version 1.42.0 to 2.8.0

15.16.0 - Errata

  • Facebook APIs are currently undergoing major reviews and changes which are affecting the ability of newly created Facebook application IDs to access the posts and events commonly presented in search results by Funnelback. Further updates or guidance will be issued to address these issues when Facebook makes it possible for Funnelback to do so.

Funnelback 15.14.0

Released : 22 March 2018

Supported until: 22 March 2021 (Long Term Support Version)

15.14.0 - New features

  • Administration of metadata mapping and xml indexing has been overhauled. -- Adds more structure when defining metadata classes and sources for HTML and XML. -- Provides suggestion of common metadata sources within the target index. -- Adds clear inline help when configuring XML indexing options.
  • Underlying APIs for metadata mapping and xml indexing have been introduced.
  • Replaced web crawler HTTP processing, adding support for a range of modern HTTP protocol options, including gzip compression.

15.14.0 - Selected improvements and bug fixes

  • Accessibility Auditor now fetches documents for auditing via the same HTTP mechanism as the web crawler, allowing for form interaction, SSL and proxy settings to be applied.
  • Display of metadata fields in search results no longer requires the 'both' summary mode. Requested 'SF' summary field metadata is now returned regardless of the 'SM' summary mode.
  • Allow for content type and additional headers to be set per profile.
  • Fixed a number of errors in calculations for date facets.
  • Reduced memory usage in a number of components (e.g. metaspace when groovy scripts are reloaded, large padre packet compression in-memory).
  • Removed the accessibility auditor G4 check, which produced unhelpful noise on all documents.
  • Introduced a configurable limit on user-requested requested num_ranks - ui.modern.external_num_ranks_limit.
  • Unauthenticated users can now be blocked from preview profiles using the restrict_preview_to_authenticated_users option.
  • Facets are now accessible by name in groovy and freemarker templates by calling the getFacetByName method on the SearchResponse object for example transaction.response.getFacetByName("facet-name").
  • Social media custom templates are now configured via collection.cfg options, and have new metadata mappings.
  • The live URLs of documents from database and directory collections no longer refer to a legacy cgi script. The document's cache URL is now used.
  • Tracking of user click actions no longer relies on HTTP referrer headers.
  • Fixed problems in handling of default ports when canonicalising URLs.
  • Mediator PullLogs command now transfers sub-directories of logs recursively.
  • Removed some cases where XML documents were incorrectly treated as HTML.
  • Added support for Server Name Indication (SNI) when fetching SAML metadata.
  • Improved SAML configuration to allow independent configuration between admin and search.
  • Improved document title fixer to eliminate some additional undesirable titles.
  • Introduced post_collection_create_command setting to simplify initial collection creation in multi-server environments.
  • Introduced debugging API call for investigating crawler errors in form interaction.
  • Improved performance of query processing when very large numbers of metadata mappings are used.
  • Introduced an experimental option to prevent concurrent in-crawl form interaction - crawler.allow_concurrent_in_crawl_form_interaction.
  • Upgraded Freemarker library to 2.3.27 (from 2.3.25) which provides some new template syntax - See http://freemarker.org/docs/versions_2_3_27.html.
  • Upgraded to Tika 1.17 (from 1.10) which provides improved file filtering support - See https://github.com/apache/tika/blob/1.17/CHANGES.txt.
  • The JSONToXML filter no longer treats the JSON key content as a special value.

15.14.0 - Configuration Upgrade Steps

The following changes will be automatically performed on all configurations during the upgrade process. Configurations migrated from older versions after the upgrade will need to have update-configs.pl manually run to apply these changes.

  • Users with access to metamap.cfg and xml.cfg in Funnelback's file manager will be granted the new sec.metadata-mapping and sec.xml-index permissions.
  • Content Auditor metadata configuration will be set in collections containing an an existing metamap.cfg file to avoid them inheriting the new default long metadata name based configuration.
  • metamap.cfg files will be migrated to the new metadata-mapping.cfg format.
  • xml.cfg files will be migrated to the new metadata-mapping.cfg and xml-index.cfg formats.
  • Invalid lines in profile.cfg files (possibly from padre-arg-sw) will be removed.

15.14.0 - Upgrade Issues

  • Web crawl NTLM authentication no longer uses the same config settings as HTTP basic authentication. Web collections relying on NTLM authentication must be reconfigured to use the new crawler.ntlm.domain, crawler.ntlm.username and crawler.ntlm.password configuration settings.
  • Since the default summary field metadata value (i.e. -SF query processor option's default of [a,c,p,s]) is now visible without -SM being set, search frontends which set no SF vaule will return metadata for the a, c, p and s metadata classes.
  • The search interface no longer allows unnecessary activation of J2EE sessions and ui.modern.session.set_userid_cookie is no longer used - Cookies are automatically used as required if search session tracking is enabled.
  • Crawler Javascript link extraction is now off by default. Any collections relying on the old 'on' default must have it set directly during upgrade.
  • Support for instant-update style feeds (handle-feed.cgi) has been removed - Push collections are recommended as a replacement.
  • The workflow publish hook is no longer called with both the preview and live profile files. Instead it will always be called with one file path each time it is run. The hook now runs each time a profile file is edited in the classic admin UI and each time any config file is edited via the admin API this includes the marketing UI and implementer UI.
  • The JSONToXML filter has been updated such that it makes a better attempt at producing valid XML this can result in xml element names being modified as well as some characters being stripped from the content.

15.14.0 - Upcoming changes

  • A future version of Funnelback will remove the $SEARCH_HOME/lib/java/all directory in favour of using a new layout. If you are currently accessing this directory with workflow scripts or similar, you are encouraged to transition away from doing so, and to contact us to discuss any cases where transitioning is technically difficult.

Funnelback 15.12.0

Released : 10 November 2017

15.12.0 - New features

  • Overhauled faceted navigation, greatly simplifying the data-model for rendering, and simplifying implementation of many previously complex scenarios.
    • Adds support for checkbox facets with results counts.
    • Adds support for tabbed presentation, radio button facets and mixed facet category sources.
    • Introduces a new configuration interface including a preview page and troubleshooting tips.
    • Adds support for facets based on queries, numeric ranges, and collections.
    • Substantially improved performance of metadata-based facet queries.
  • Expanded the range of checks performed by Accessibility Auditor, and improved the reporting interface.
    • A new reporting overview provides summaries of changes by WCAG levels.
    • Auditing of a single document now breaks down issues by WCAG principles.
    • Introduces additional reporting summaries on each WCAG technique and success criterion.
    • Enables filtering and csv exporting of the Accessibility Auditor reports.
  • Introduced option of x.509 client certificate authentication for search interface.
  • Introduced experimental support for SAML authentication in search and administration interfaces.
  • Introduced support for gathering content from HPE Content Manager 9.1.
  • Added an all results endpoint for streaming back all results of a query, even if the set is very large.
  • A non-expiring application token is now supported by most Funnelback APIs. See API Token Authentication.
  • Introduced new auto-completion plugin.
  • Introduced an advanced update option to reapply any reconfigured gscopes a collection's live view.

15.12.0 - Selected improvements and bug fixes

  • Improved compression of indexes when push collection replication is used.
  • Increased the number/total-size of unique terms a search index may contain, and improved handling of very frequent terms.
  • Improved efficiency of classic administration interface with large numbers of collections.
  • Allowed for a number of query-time settings to be set within a service (profile.cfg) rather than only at the collection level.
  • Introduced ui.modern.padre_response_size_limit_bytes limit on padre response size to avoid large queries consuming all query processor memory.
  • Improved jetty request logging to limit access log size, and compress on rotation.
  • Fixed handling of multiple metadata items in external metadata when facet_item_sepchars is used.
  • Worked-around web servers returning content gzipped even if it is not requested.
  • Improved efficiency/reliability of groovy script change detection.
  • Improved push collection snapshot APIs and marking of incomplete/failed snapshots.
  • Improved fidelity of queries reported in analytics by eliminating unnecessary query simplifications.
  • Imposed a limit on JVM metaspace usage to ensure it is collected regularly.
  • Fixed form interaction to remove expired cookies and use defaults for form 'action' and 'method' parameters.
  • Increased the default values for max download size and max parsing size of file to 10MB.
  • Fixed an issue where the Modern UI cached view would not process all documents filtered to XML or left as XML (detected by Content-Type) as XML with the expected XSL transformation.
  • Custom collections, Database collections and Directory collections now support filtering. To enable filtering in existing custom collections a raw bytes store must be used see Custom Collections - Cache copies don't work
  • Fixed a bug where the modern UI would return all profile configurations in the data model, rather than just the active one.
  • Gscopes are no longer referenced by bit number, instead they are given names like metadata classes. Where bit numbers are still used, they will now be interpreted as the gscope name (e.g. a gscope named '5').
  • Introduced support for auto-expansion of the number of available gscope bits.

15.12.0 - Upgrade Issues

  • Faceted navigation has been improved to support in built sorting, checkbox facets, tab facet and an easier to use facet data model. To take advantage of this you may need to upgrade your facets. To do it please follow this guide.
  • The faceted_navigation.date.sort_mode option is deprecated and will only work with legacy date facets. Built-in facet sorting can now be used instead.
  • Funnelback now includes an improved auto-completion system called Concierge. To upgrade existing collections please follow the upgrading to concierge guide.
  • Funnelback's data-api has been removed. Admin-API provides equivalent calls, however please note that endpoints and JSON envelopes have changed.
    • /admin-api/collection-info/v1/collections/collection_name/url/data replaces /data-api/v1.0/urls/info
    • /admin-api/accessibility-auditor/v1/ replaces /data-api/v1.0/wcag/
    • /admin-api/predictive-segmentation/v1/ replaces /data-api/v1.0/predictive-segmentation
  • The deprecated features text-miner, classic wca-reporter, and classic analytics have been removed from the default installation.
  • Configuration file headers (containing the full path of the file) have been removed.
  • Jetty access log filenames have changed - Any systems reading them based on their old filenames may need to be updated.
  • Jetty access logs are now rotated daily or when they reach 512MB in size and kept for either 90 days or until the total size reaches 1.5GB. See configuring embedded web server to customise this.
  • URL fill facets behaviour has been updated so that facet values will be returned for every parent folder of the currently selected folder (in addition to returning the child folders of the current path) to be consistent with other facet types where the selected categories are present in the data model. For example when drilling down to folder1/folder2/folder3/ facets values will now be returned for folder1, folder2, folder3 all with their selected flag set to true. The default <@s.Category> macro has been updated accordingly and should result in no actual change in display. If custom FreeMarker facet macros are used they will need to be updated to pay attention to the new facet values.
  • Contextual Navigation has been updated so that no "site" cluster will be returned if there's only 1 site in it. As a result response.resultPacket.contextualNavigation.categories may be empty if there are no "topics", no "types", and the "site" cluster contain only 1 site. This change is compatible with the default FreeMarker tags for Contextual Navigation, no change is required when using them. Custom FreeMarker Contextual Navigation tags may need to be updated accordingly.
  • The behaviour of search facet parameter facetScope has changed such that values set there no longer override parameters set on the URI e.g. facetScope=x%3Dfoo&x=bar results in x being set to both foo and bar rather than just foo.
  • The version of groovy included has been upgraded from 2.3.7 to 2.4.12.
    • Please note that the new version changes the precedence of cast operations - See https://issues.apache.org/jira/browse/GROOVY-5185
  • Funnelback now bundles Bootstrap version 3.3.7, update the path to resources in FTL forms from ${GlobalResourcesPrefix}thirdparty/bootstrap-3.0.0/ to ${GlobalResourcesPrefix}thirdparty/bootstrap-3.3.7/. Note that v3.0.0 will be removed from Funnelback in a future release.
  • The version of java included with Funnelback has been updated to 8u141. Please note that some insecure SSL certificates will no longer be accepted by the new version.
  • The query processor option num_ranks has changed behaviour when set to 0. When set to zero it no longer skips query processing and instead will behave the same as positive values for num_ranks except no results will be displayed.
  • The type of gscopesSet within a Result returned by the modern UI has changed from a Set of Integer to a Set of String to reflect that gscopes are accessed by name rather than by bit number.
  • Collection.cfg option gscopes.other_bit_number has been renamed to gscopes.other_gscope, Funnelback remains compatible with the old key.
  • Database and Directory collections no longer use the XML store set by store.xml.class and instead store XML documents into a raw bytes store set by store.raw-bytes.class. Records are no longer stored using the primary key and instead are stored using the same URI that is set in the <funnelback_url> element, these collections no longer require that xml.cfg map the document URL. Existing Database and Directory collections require that the collection be updated before cache copies will work.
  • Raw bytes store com.funnelback.common.io.store.bytes.FlatFileStore has been fixed such that some URLs no longer cause issues with the store, as part of this the store is no longer compatible with previous versions. Any collections using this store should have a full update run and if possible switch to using com.funnelback.common.io.store.bytes.WarcFileStore.
  • The faceted_navigation.date.sort_mode option is deprecated and will only work with existing date facets, the build facet sorting can be used instead.
  • Faceted navigation has been improved to support in built sorting, checkbox facets, tab facet and an easier to use facet data model. To take advantage of this you may need to upgrade your facets. To do it please follow this guide.
  • To upgrade auto-completion to use new concierge auto-completion plugin please follow this guide.

Funnelback 15.10.0

Released : 12 April 2017

15.10.0 - New features

  • Redesigned accessibility auditor reporting interface, and new WCAG technique implementations.
  • Redesigned trend alerts reporting interface.

15.10.0 - Selected improvements and bug fixes

  • Curator rule editing now supports custom structured parameters for most actions.
  • Introduced a minimum length (15 characters) when user change their password.
  • The default hashing mechanism for passwords is now BCRYPT. Existing user passwords will continue to use the legacy hashing mechanism until the password is updated via the Administration interface or accounts API.
  • Session information is now correctly stored for very long queries.
  • Added support for configuring acceptable SSL cipher suites and protocols.
  • Web crawler will no longer report 'invalid' protocols like data: and tel:.
  • Introduced standardised filters for converting CSV or JSON to XML.
  • Fixed overwriting of changed cookie values during form interaction authentication.
  • Improved URL drilldown selection within content auditor.
  • Introduced profile management interface.
  • Introduced a default timeout (50 seconds) for IncludeURL tags, avoiding blocking threads forever in some cases.
  • Fixed cases where collection update progress would not be displayed after starting an update.
  • Fixed date sorting to work correctly for past and future documents (rather than being based on proximity to current date).
  • The query processors (padre-sw) -SF option now accepts regex.
  • The version of Jsoup included has been upgraded to 1.10.2.
  • Log files are no longer created when collections which do not exist for the current Funnelback installation are requested.

15.10.0 - Upgrade Issues

  • Due to the improved password hashing mechanism, HTTP Basic Authentication for APIs is considerably slower than in previous versions. If performance is a concern, API users are advised to switch to token-based authentication.
  • The sessions database schema has changed to allow for longer queries. The inbuilt session databases will be automatically upgraded, however if you are using a external database for sessions you will need to run update-session-db.groovy with the appropriate driver and url.
  • Due to the new IncludeURL default timeout, IncludeURL calls expected to take longer then 50 seconds should manually set a higher timeout.

15.10.0 - Notice

  • Please be aware that 15.10 is the last version of Funnelback which will support running on Windows Server 2008.

Funnelback 15.8.0

Released : 16 December 2016

15.8.0 - New features

  • Introduced a redesigned facet editing interface.

15.8.0 - Selected improvements and bug fixes

  • Introduced support for HP RM 8.2+
  • Introduced query-gscopes.cfg configuration file allowing gscopes to be set based on a query (without the need for facets).
  • Removed legacy 'htpasswd' files and associated scripts. This fixes an issue where password resets could be overwritten by subsequent user password changes.
  • Improved the compatibility of component collections in meta collections. Padre indexes are no longer made incompatible by varying gscopes, number of metadata classes or the type of metadata classes.
  • Padre recency ranking feature is now based on proximity to the current date, in general the closer the date the higher the score.
  • A new filtering framework has been introduced that is more flexible while simpler to implement.
  • Push has V2 API content calls which allow for multiple values per metadata key.
  • Push PUT document API calls are now able to take even more advantage of multiple CPUs.
  • Implemented a limit on image scaling input to prevent potential OOM situations.
  • Errors from Funnelback's search interface are now logged per collection (under web/log/ on windows and data/coll-name/log elsewhere) rather than in one shared log file.
  • Addressed a number of cross site scripting security issues.
  • Restructured product documentation, and improved the graphical design.
  • Facet template tags are now able to impose a display order on facets using orderedNames attribute.
  • Upgraded Freemarker library to 2.3.25 which provides some new template syntax - See http://freemarker.org/docs/versions_2_3_25.html.
  • Introduced ability to compare curator configurations between live and preview.
  • Fixes/improvements to the web crawler's form interaction support.
  • Migrated the functionality of a number of mediator (bin/*.pl) Perl scripts to Java.
  • Fixes a issue where perl publish hook scripts would not be correctly called for curator and synonyms on Windows.
  • Fixed an issue with collections not displaying the correct "last update" date

15.8.0 - Upgrade Issues

  • IsolatedFilterProvider is no longer supported. If this filter is used, filtering will still occur, but will not be isolated in a separate process.
  • FileTypeMappedFilterProvider is no longer supported.
  • JPEGJavaFilter and RTFJavaFilter are no longer provided. TikaFilterProvider is covering JPEG and RTF formats.
  • LotusFilterProvider and TextCleanupFilterProvider are no longer provided.
  • AbstractFilterProvider constructor parameters have changed. Previously (String collectionName, boolean inlineFilterEnabled) is now (File searchHome, CollectionId collectionId). If classes were inheriting from AbstractFilterProvider their constructor will need to be changed accordingly.
  • Post gather filtering (convert.pl) is no longer supported, all filtering must be done inline.
  • Windows now uses symlinks for live/offline views. Existing collections will have symlinks created on upgrade. The Windows user which runs scheduled updates and users which run updates from the command line will need the permission SeCreateSymbolicLinkPrivilege.
  • The JAVA_OPTS environment variable will no longer be respected by groovy commands in pre/post workflow commands.
  • Instant update delete for Filecopy collections no longer require encoding of the system URLs supplied similar to instant update add e.g. \\example\sp%20ce should be supplied as \\example\sp ace.
  • funnelback-reporting.jar has been moved from $SEARCH_HOME/bin/ to $SEARCH_HOME/lib/java/all/.
  • Image scaling is now limited to input images less than 1MB by default. Set default_image_fetcher.max_source_image_bytes to a larger value in global.cfg during upgrade if required.

Funnelback 15.6.0

Released : 27 June 2016

15.6.0 - Selected improvements and bug fixes

  • Introduced Translucent Document Level Security, which allows for some information to be exposed about documents the current user is not permitted to see.
  • Renamed Funnelback's "Query Completion" feature to "Auto-complete".
  • Introduced a new Custom Servlet Filter Hook mechanism to allow for advanced pre/post filtering of search requests.
  • Fixed a bug preventing users from copying an in-live-only synonym/best-bet/curator.
  • Fixed an issue with the breadcrumb missing "tuning" link in Tuning History page.
  • Fixed bug in accessibility checker where StackOverflowErrors would be generated and stored in the log.
  • Fixed bug in accessibility reporter where checking documents with large numbers of errors would result in an OutOfMemoryError.
  • Funnelback's installer is now 64-bit, and no longer requires 32-bit compatibility libraries in order to install.
  • Crawls will now exit with a success status if they store documents (regardless of whether they are downloaded or copied-forward by incremental crawling).
  • Several design/interaction improvements and enhancements to the Admin, Documentation, SEO Auditor and Content Auditor interfaces.
  • Push collections will now accept header metadata with the prefix X-Funnelback-Push-Meta-Data- - The old prefix with underscores is still supported, but it discouraged as some proxy servers strip such headers by default.
  • Implemented ability to compare live and preview versions of curator rules.
  • Corrected handling of spaces in filenames within web resources.

15.6.0 - Upgrade Issues

  • Please note that the renaming of "Query Completion" to "Auto-complete" affects a large number of collection.cfg settings as well as csv file name and a number of other areas. Where possible, the installer will automatically rename settings, file and update the relevant setting references in ftl files. Custom workflow scripts interacting with these settings or files may need to be manually updated while upgrading Funnelback.
  • If classic-ui is installed, database collections using the classic-ui's serve-db-document links may no longer work. Cache views, ideally the modern-ui's cache, should be used to provide links for database records instead.
  • The systems for starting most gathering components have been changed - While no functionality should be affected, please be aware that the format of the update logs have been changed.
  • The modern-ui cache controller applies a stricter security model for cache copies and any collections which have a Security field enabled will not be able to serve cache copies. Previously FileCopy collections had a security field set by default, even if DLS was not enabled, and unless removed this will prevent cache copies from being accessible after upgrading.
  • When upgrading, the installer will now move leftover files from lib/java/all to lib/java/prev-timestamp. This is to prevent custom or patched .jar files (deployed in earlier versions) from interfering with the upgraded systems. Files in lib/java/prev-timestamp should be inspected and moved back into lib/java/all after the upgrade manually if they are required.

Funnelback 15.4.1

Released : 29th April 2016

15.4.1 - Selected improvements and bug fixes

  • Fixed a bug preventing the collection links in the collection overview from switching the selected collection.
  • Fixed a bug preventing older versions of Internet Explorer from logging into Funnelback administration.
  • Fixed issues arising from patch jar files being left in lib/java/all after upgrade - Any unknown files will now be moved to lib/java/previous-(timestamp) during upgrade.
  • Fixed problem of search box capturing keyboard events on documentation pages.
  • Fixed an issue where Push collections in slave mode could not be switched to master (DEFAULT) mode unless master could be reached.
  • Fixes an issue where the Admin UI is unable to prepare Funnelback for upgrade.

Funnelback 15.4.0

Released : 14th April 2016

15.4.0 - Selected improvements and bug fixes

  • Improved Modern UI logging: FreeMarker error are now logged to the Modern UI log rather than the Jetty log, and extra searches messages now contain the proper collection and profile information.
  • The classic admin UI now uses the same login page and authentication system as the marketing dashboard.
  • Funnelback now includes an output connector for ManifoldCF version 2.x, in place of including a full installation of ManifoldCF v1.x.
  • Push collections use a better algorithm for choosing which generation to merge.
  • Push multi server setups use significantly less bandwidth and less disk IO for replication.
  • Push multi server setups can now ignore redirect files on the index.
  • Push will now re-index large generations with mostly killed documents.
  • Push multi server setups now support compression and no longer require the webdav service.
  • Push supports parallel indexing.

15.4.0 - Upgrade Issues

  • Deprecated binaries have been removed from SEARCH_HOME/linbin/ (Linux) and SEARCH_HOME/wbin/ (Windows): info-zip, libxslt, modssl, nginx, openssl, xsltproc. These will need to be re-installed separately if needed.
  • Since Funnelback no longer includes an embedded Manifold CF installation, please ensure ManifoldCF is installed as described by connecting enterprise repositories if it is to be used.
  • Funnelback's embedded Jetty web server no longer provides JSP support for web applications. This was only used by ManifoldCF which now needs to be installed separately.

15.4.0 Errata

  • An issue exists with the Admin UI's "Prepare Funnelback for upgrade" system menu option in 15.4.0. To work around this issue, please use the following Push API call directly

    POST /v1/upgrade/prepare

An interface to access this call directly is available within Funnelback's UI (https://<host name>:<admin port>/search/admin/api-ui/ ) under the Push API tab in the push-api-collection section.

Funnelback 15.2.0

Released : 9th March 2016

15.2.0 - Selected improvements and bug fixes

  • Renamed Modern Admin UI to Administration Interface (in general) and Marketing Dashboard (for specific marketing focused functionality). Note that both are distinct from the older Classic Admin UI.
  • Curator can now be configured to examine additional URL parameters with the ui.modern.curator.query-parameter-pattern setting.
  • Synonym blending will now run on complex queries.
  • The query processor will execute queries when the query parameter is not set and the system query s is set.
  • Push will now correctly read the correct worker thread count config option from push.worker-thread-count rather than from worker-thread-count.
  • Fixed an issue where duplicate pagination within content auditor would persist even after leaving the duplicate area.
  • Improved styling of the documentation, content auditor and the administration interfaces.
  • Added support for iframe tags within best bet previews.
  • Fixed default search template to display curator driven best bets.
  • Improved html tag boundary sentence detection within content auditor's readability grade calculation.
  • Fixed result collapsing presentation for result pages after the first.

15.2.0 - Upgrade Issues

  • The Administration Interface now respects additional permissions for synonyms (sec.synonym), best bet (sec.best-bet) and curator (sec.curator). These permissions will be granted to users with the "administrator" and "editor" roles on upgrade. These permissions grant complete access to the corresponding feature (view & modify). If custom file manager rules were previously configured in the users INI files to control specific permissions on corresponding configuration files (e.g. synonyms.cfg), the users INI file will need to be reviewed to add or remove the new permissions as needed.
  • The modernui.log used to contain logs for both the public and admin context. It is now split two separate log files modernui.Public.log and modernui.Admin.log.

Funnelback 15.0.0

Released : 22nd January 2016

15.0.0 - New features

  • Marketing Dashboard Administration Interface (originally called Modern Admin UI) providing an improved administration interface for common search service management tasks. Specific improvements include:
    • Improved reporting interface, including mapping and time based comparisons.
    • Improved editing interface for best bets, including bets bet previewing.
    • Improved editing interface for curator rules, including a number of new triggers.
    • Improved editing interface for synonyms and search tuning data sets.
    • Ability to publish individual items (such as a best bet) rather than entire configuration files.
    • Better user session management, including the ability to logout without closing the web browser.
    • Introduces a 'service' concept to separate collections purely for back-end gathering purposes from those expected to receive user queries.
  • Structured administration API covering all Administration Interface (originally called Modern Admin UI) features.
  • Content Auditor 'recommendation' reports:
    • Reading Grade scores for textual content.
    • Missing Metadata identification.
    • Duplicate Title identification.
    • Date Modification reporting.
    • Response Time reporting.
    • Undesirable Text identification.
    • Duplicate Content identification.
  • Push collections now support replicating their indexes to remote query processor servers.
  • Custom filters can now be implemented to receive a parsed HTML DOM model rather than raw bytes/characters using the JSoup Filter interface.
  • TRIMPush collections now support HP Records Manager v8 (formerly TRIM). TRIM v7 is still supported, but previous versions (v6.2, v6.0, v5) are not.
  • Form interaction now supports multiple 'in-crawl' rules, and the form action targets may be defined by regular expressions.

15.0.0 - Selected improvements and bug fixes

  • Funnelback now supports running the public search interface over HTTPS by default - Simply configure the desired HTTPS port during installation.
  • Introduced a new curator trigger which can perform a range of string and numeric comparisons on an arbitrary URL parameter.
  • Introduced a new curator trigger which can trigger when specific facet categories are selected.
  • Introduced numeric comparisons for curator segment triggers.
  • Upgraded API/library versions for several social media sources.
  • Fetching URL information is now isolated from the web server process to reduce the impact of errors during fetch operations.
  • Improved efficiency of duplicate detection within Content Auditor.
  • search.json and suggest.json now return CORS headers to allow cross-origin requests.
  • Modern UI hook scripts now include $SEARCH_HOME/lib/java/groovy and the collection's @groovy directories on their class path when they are run, allowing for common functions to be stored there and reused.
  • Funnelback now includes a JSON Schema definition of the search.json output to allow for schema comparison between versions.
  • Upgraded filtering libraries, providing fixes for a number of PDF extraction problems and many other filtering issues.
  • Improved logging context and consistency of configuration.
  • Fixed handling of long metadata classes used for security when merging indexes.
  • Fixed handling of long metadata classes beginning with 'd' and 't'.
  • Fixed handling of long metadata classes when using search session features.
  • More robust handling of query/click logs when building search analytics.
  • Improved configuration of matrix connection when using document level security.
  • Improved management of temporary/working files when running Funnelback services.
  • License limit information is now available within the (modern) admin UI.
  • Reduced memory usage of some social media gathering operations.
  • Avoid double-wrapping of jsonp search results which occurred in some configurations.
  • Fixed colfield indexer option operation on push collections.
  • Improved push collection debugging and recovery features.
  • Improved a number of cases where invalid HTML was made worse during filtering/accessibility checking.
  • Updated to match new Funnelback logos and branding.
  • Removed constraint on meta collection components requiring matching ordering in metamap.cfg and xml.cfg.
  • Updated Freemarker templating library to 2.3.23.

15.0.0 - Upgrade Issues

  • Some features from Funnelback's Classic Admin UI have been removed where they overlap with the Administration Interface (originally called Modern Admin UI). Services will need to be manually created for relevant collections to allow these features to be used.
  • Funnelback's Classic search UI is no longer included with the standard Funnelback install. Where required, a separate install package is available to provide Classic Search UI compatibility under Jetty, however Classic Search UI under IIS is no longer supported.
    • As a result of the removal of Classic Search UI, certain static resources used by some Modern UI templates have moved from /search to /s/resources-global. Most references will be automatically fixed during installation, but manual fixes may be required in some cases.
    • Templates referring to search/help/simple_search will need to be updated, as this generic documentation for classic UI is no longer included. Relevant help content could be added to a collection's web resources folder instead.
  • Administration Interface best bets are implemented using curator rules rather than the historical mechanism. Existing search templates may need to be updated to ensure the resulting adverts are styled as desired.
  • The format of synonyms.cfg files has changed to include some additional id and edit-time properties. Files using the old format will still be read, however only the new format will be written.
  • Funnelback administration no longer supports Internet Explorer version 9 or below - Please ensure all administrators have up-to-date IE versions prior to upgrading.
  • TRIMPush collections: The name of the user defined fields has changed from udfX (where X was the field number) to the actual name of the field, e.g. Cost, Category, etc. If you are using user defined fields you will need to update your metadata mappings to account for this change. See properties and user fields to obtain the new name of the user fields.
  • Late binding security check scripts invoked during query processing are no longer provided. Implementing early or late binding checks in native code triggered using the secPlugin query processor option is the recommended replacement.
  • The form Modern UI URL parameter cannot be passed blank anymore. It can still be omitted entirely (resulting in the default simple form being used) but passing it blank (i.e. http://server/s/search?collection=c&form=) will result in an 400 Bad Request HTTP response.
  • The Admin API authentication scheme has changed from HTTP Basic to a token based system. Client applications needs to be updated.
  • As Funnelback now supports a per item publication model, all curator rules will be enabled on upgrade. Any rules you do not wish to be enabled should be unpublished removing them from the live search.
  • Jetty web server context directories have been renamed from $SEARCH_HOME/web/conf/contexts-https and $SEARCH_HOME/web/conf/contexts-http to $SEARCH_HOME/web/conf/contexts-admin and $SEARCH_HOME/web/conf/contexts-public
  • Form Interaction: values are now expected to be URL-encoded in form_interaction.cfg, to allow for characters like '&' to be part of passwords. An entry like username=admin&password=&min should be written as username=admin&password=%26min instead.
  • sec.content.optimiser has been renamed to sec.seo-auditor.
  • The search help pages at /search/help/simple_search and /search/help/query_language_help are not available anymore. Form files that point to them should be updated to remove the links.
  • Any analytics database initially created in Funnelback version 9 must be updated from scratch.
  • The following features have been removed - Funnelback Shell, Store Service (superseded by push collection API), File Transfer (superseded by multi-server WebDAV support), EAS (superseded by nte new Administration Interface), Late binding security (superseded by early binding security, which can support plugable late binding checks).
  • SEO Auditor is now available only as part of the Administration Interface, and can no longer be configured for public access.
  • Gathering of TRIM v6 sources is no longer supported (superseded by v7 and v8 support).
  • The authentication mechanisms for the Admin API have been changed to require a login call rather than HTTP Basic headers. See API UI for information on handling the new authentication scheme.

For older, no longer supported, versions please see the Historical Release Notes page.

top

Funnelback logo
v15.18.0