Mesh configuration properties

This page describes the configuration properties that can be used to control behavior in Bitbucket Mesh. Create the mesh.properties file, in the home directory, and add the system properties you need, use the standard format for Java properties files.

Bitbucket Mesh must be restarted for changes to become effective.

Default values for system properties, where applicable, are specified in the tables below.

Authentication

Default valueDescription
authentication.allowed-clock-skew
2m

Defines the clock skew allowed when validation expiry for signed tokens during authentication. This accounts for clock drift between Bitbucket and Mesh nodes. This value is in SECONDS unless a unit (s, m, h, d) is specified.

authentication.expiry-interval
30s

Defines the amount of time a signed token is valid after it's issued. This only affects outbound requests, such as replication requests from one Mesh node to another. Since tokens are generated right before the RPC they will be used to authenticate, this interval should generally be short. This value is in SECONDS unless a unit (s, m, h, d) is specified.

Commits

Default valueDescription
commit.message.max
262144

Controls the maximum length of the commit message to be loaded when retrieving one or more commits. Commit messages longer than this limit will be truncated. The default limit is high enough to not affect processing for the general case, but protects the system from consuming too much memory in exceptional cases.

commit.message.bulk.max
16384

Controls the maximum length of the commit message to be loaded when retrieving commits in bulk. Commit messages longer than this limit will be truncated. The default limit is high enough to not affect processing for the common case, but protects the system from consuming too much memory when many commits have long messages.

Database

Default valueDescription
database.pool.idle-size
5

Defines the number of connections the pool tries to keep idle. The system can have more idle connections than the value configured here. As connections are borrowed from the pool, this value is used to control whether the pool will eagerly open new connections to try and keep some number idle, which can help smooth ramp-up for load spikes.

database.pool.max-size
50

Defines the maximum number of connections the pool can have open at once.

database.pool.connect-timeout
15

Defines the amount of time the system will wait when attempting to open a new connection before throwing an exception. The system may hang, during startup, for the configured number of seconds if the database is unavailable. As a result, the timeout configured here should not be generous.

If no unit is specified (ms, s, m, h) this value is in seconds.

database.pool.idle-timeout
30

Defines the maximum period of time a connection may be idle before it is closed. In general, generous values should be used here to prevent creating and destroying many short-lived database connections (which defeats the purpose of pooling).

If no unit is specified (ms, s, m, h) this value is in minutes.

database.pool.leak-timeout
0

Defines the maximum period of time a connection may be checked out before it is reported as a potential leak. By default, leak detection is not enabled. Long-running tasks can easily exceed this threshold and trigger a false positive detection.

If no unit is specified (ms, s, m, h) this value is in minutes.

database.pool.lifetime-timeout
60

Defines the maximum lifetime for a connection. Connections which exceed this timeout are closed the first time they become idle and fresh connections are opened.

If no unit is specified (ms, s, m, h) this value is in minutes.

Executor

Controls the thread pool for general asynchronous processing.

Default valueDescription
executor.max-threads
${scaling.concurrency}

Controls the maximum number of threads allowed in the common ExecutorService. This ExecutorService is used by for background tasks. When more threads are required than the configured maximum, the thread attempting to schedule an asynchronous task to be executed will block for up to executor.timeout until a thread in the pool becomes available. By default, the pool size scales with the number of reported CPU cores. Note: A minimum of 4 is enforced for this property. Setting the value to a lower value will result in the default 4 threads being used.

executor.timeout
60

Controls how long a thread submitting a task to the common ExecutorService should block when all threads in the pool are busy before giving up and rejecting the task.

This value is in seconds.

Git

Default valueDescription
git.path.executable

Defines the path to the git executable. If no value is set, the system searches the runtime user's PATH and standard system paths to find a suitable git executable.

Setting a value here is usually not necessary. This is left here purely for documenting how to set an explicit path.

Git Hooks

Default valueDescription
hooks.callback.timeout
240

Controls the maximum time a git hook callback is allowed to execute before being aborted, thereby rejecting the request if the hook can reject the request. When one or more refs are updated and a pre-receive or post-receive request is sent back to the RPC client, this setting controls how long the application will wait for the RPC client to respond.

This value is in seconds.

Home directory layout overrides

Default valueDescription
layout.caches-dir

Controls the directory used to store caches, including the pack-objects caches used to cache clones. Files in this directory do not need to be retained across restarts. In cloud deployments, it's perfectly acceptable to use an ephemeral drive for the caches directory. Using a drive that provides good read performance for the caches directory will greatly benefit clone performance.

Use an absolute path when configuring this property.

layout.tmp-dir

Controls the directory used to store temporary files. Files in this directory do not need to be retained across restarts. In cloud deployments, it's perfectly acceptable to use an ephemeral drive for the tmp directory.

Use an absolute path when configuring this property.

Hook Scripts

Default valueDescription
hookscripts.gc.interval
24

Defines the number of hours to wait between garbage collection runs for hook scripts.

This unit is in hours. The default is 24 hours (1 day).

hookscripts.gc.prune
168

Defines the minimum age of a hook script before it can be considered for garbage collection.

This unit is in hours. The default is 1 week.

Hosting

Default valueDescription
hosting.allow-filter
true

Controls whether partial clones, using --filter, are allowed. Partial clones are not cached, and some of the filters offered by Git can be very resource-intensive for the server to apply, so it can sometimes be more efficient to use a normal clone (or a shallow one) instead. Partial clones are enabled by default.

hosting.atomic-initial-push
true

Controls whether initial pushes are treated as an atomic push by the system. This option is enabled by default because it significantly reduces the overhead of coordinating and replicating the initial push. Only disable this option in case of issues with initial pushes.

Hosting Cache (Pack Cache)

See Scaling for Continuous Integration performance for more information about configuring the Git Cache.

Note: The settings controlled by these properties can be dynamically updated using REST. The REST configuration takes precedence over the configuration in mesh.properties.

Default valueDescription
hosting.cache.expiry.check.interval
30

Controls how frequently expired cache entries are invalidated and removed from the cache.

This value is in seconds.

hosting.cache.eviction.hysteresis
1073741824

When eviction is triggered the amount of disk space requested for eviction is calculated as (eviction.hysteresis + eviction.trigger.free.space - free space under <Mesh home directory>/caches)

This value is in bytes.

hosting.cache.eviction.trigger.free.space
6442450944

Controls the threshold at which eviction is triggered in terms of free space available on disk (specifically under <Mesh home directory>/caches)

This value is in bytes.

hosting.cache.minimum.free.space
5368709120

Controls how much space needs to be available on disk (specifically under <Mesh home directory>/caches) for caching to be enabled. This setting ensures that the cache plugin does not fill up the disk.

This value is in bytes.

hosting.cache.pack.enabled
true

Controls whether caching is enabled for git-upload-pack (clone operations).

hosting.cache.pack.maxCount
20

The maximum number of upload-pack cache entries to retain per repository. If there are more than this configured limit, the least recently accessed entry will be invalidated.

hosting.cache.pack.ttl
14400

Controls how long the caches for clone operations are kept around when there no changes to the repository.

Caches are automatically invalidated when someone pushes to a repository or when a pull request is merged.

This value is in seconds.

hosting.cache.protocol.http.enabled
true

Controls which whether caching over HTTP(S) is enabled.

hosting.cache.protocol.ssh.enabled
true

Controls which whether caching over SSH(S) is enabled.

Metrics

Default valueDescription
metrics.tags.application
Mesh

Any metrics tags that should be added to all metrics reported from this node can be added as properties under metrics.tags. For instance, if metrics reported by this node should be tagged with region=us-east-1, this node is running under, add metrics.tags.region=us-east-1

management.metrics.export.datadog.enabled
false

Datadog metrics configuration. Please set enabled=true to enable publishing of metrics to Datadog and uncomment and update the relevant Datadog configuration properties in the section below.

management.metrics.export.datadog.api-key
YOUR_API_KEY
management.metrics.export.datadog.step
30s
management.metrics.export.dynatrace.enabled
false

Dynatrace metrics configuration. Please set enabled=true to enable publishing of metrics to Dynatrace and uncomment and update the relevant Dynatrace configuration properties in the section below.

management.metrics.export.dynatrace.api-token
YOUR_TOKEN
management.metrics.export.dynatrace.device-id
YOUR_DEVICE_ID
management.metrics.export.dynatrace.uri
YOUR_URI
management.metrics.export.influx.enabled
false

InfluxDB metrics configuration. Please set enabled=true to enable publishing of metrics to InfluxDB and uncomment# and update the relevant InfluxDB configuration properties in the section below.

management.metrics.export.influx.auto-create-db
true

Whether to create the Influx database if it does not exist before attempting to publish metrics to it. (Default: true)

management.metrics.export.influx.batch-size
10000

Number of measurements per request to use for this backend. If more measurements are found, then multiple requests will be made. (Default: 10000)

management.metrics.export.influx.batch-size.compressed
true

Whether to enable GZIP compression of metrics batches published to Influx. (Default: true)

management.metrics.export.influx.batch-size.connect-timeout
1s

Connection timeout for requests to this backend. (Default: 1s)

management.metrics.export.influx.batch-size.consistency
one

Write consistency for each point. (Default: one)

management.metrics.export.influx.batch-size.db
mydb

Tag that will be mapped to "host" when shipping metrics to Influx. (Default: mydb)

management.metrics.export.influx.batch-size.enabled
true

Whether exporting of metrics to this backend is enabled. (Default: true)

management.metrics.export.influx.batch-size.num-threads
2

Number of threads to use with the metrics publishing scheduler. (Default: 2)

management.metrics.export.influx.batch-size.password
mysecret

Login password of the Influx server.

management.metrics.export.influx.batch-size.read-timeout
10s

Read timeout for requests to this backend. (Default: 10s)

management.metrics.export.influx.batch-size.retention-policy
my_rp

Retention policy to use (Influx writes to the DEFAULT retention policy if one is not specified).

management.metrics.export.influx.batch-size.step
1m

Step size (i.e. reporting frequency) to use. (Default: 1m)

management.metrics.export.influx.batch-size.uri
http://localhost:8086

URI of the Influx server. (Default: http://localhost:8086)

management.metrics.export.influx.batch-size.user-name
myusername

Login user of the Influx server.

management.metrics.export.jmx.enabled
${jmx.enabled}

Configures whether metrics should be exposed over JMX

management.metrics.export.jmx.domain
metrics
management.metrics.export.jmx.step
5s
management.metrics.export.newrelic.enabled
false

NewRelic metrics configuration. Please set enabled=true to enable publishing of metrics to NewRelic and uncomment# and update the relevant NewRelic configuration properties in the section below.

management.metrics.export.newrelic.account-id
YOUR_ACCOUNT_ID
management.metrics.export.newrelic.api-key
YOUR_API_KEY
management.metrics.export.newrelic.step
1m

The interval at which metrics are sent to New Relic. See Duration.parse for the expected format. The default is 1 minute.

management.metrics.export.signalfx.enabled
false

SignalFX metrics configuration. Please set enabled=true to enable publishing of metrics to SignalFX and uncomment# and update the relevant SignalFX configuration properties in the section below.

management.metrics.export.signalfx.access-token
YOUR_ACCESS_TOKEN
management.metrics.export.statsd.enabled
false

StatsD metrics configuration. Please set enabled=true to enable publishing of metrics to StatsD and uncomment# and update the relevant StatsD configuration properties in the section below.

management.metrics.export.statsd.host
statsd.example.com
management.metrics.export.statsd.port
9125
management.metrics.export.statsd.protocol
udp

Process execution

Controls timeouts for external processes, such as git.

Default valueDescription
process.timeout.execution
120

Configures a hard upper limit on how long the command is allowed to run even if it is producing output.

This value is in seconds. Using 0, or a negative value, disables the timeout completely.

process.timeout.idle
60

The idle timeout configures how long the command is allowed to run without producing any output.

This value is in seconds. Using 0, or a negative value, disables the timeout completely.

Profiling

Default valueDescription
profiling.enabled
${atlassian.profile.activate:false}

Controls whether profiling should be enabled or not

profiling.min-frame-time
${atlassian.profile.mintime:1}

Defines the threshold time (in milliseconds) below which a profiled event should not be reported.

profiling.min-trace-time
${atlassian.profile.mintotaltime:0}

Defines the threshold time (in milliseconds) below which an entire stack of profiled events should not be reported.

profiling.max-frame-name-length
${atlassian.profile.maxframenamelength:300}

Defines the maximum length of a profiling frame in the profiling log. Setting this value too high can lead to out of memory issues because profiling frames are kept in memory until the request ends. Longer frames will be abbreviated.

Repair

Exponential backoff between (failed) attempts to repair a repository replica

Default valueDescription
repair.backoff.initial-delay
30

Configures the delay between the first repair failure and the next attempt to repair the replica. This value is in seconds.

repair.backoff.jitter
0.05

Configures the fraction of jitter to apply to the next delay to help spread out delays of concurrent repair operations. The default is 5%, which results in the next calculated delay being increased or decreased by up to 2.5%

repair.backoff.max-delay
1800

Configures the maximum delay between two repair attempts. This value is in seconds. The default is 30 minutes.

repair.backoff.multiplier
1.2

Configures the factor by which the delay between two repair attempts should be increased The default value is 1.2, for a ~20% increase between attempts (+/- jitter).

repair.max-attempts
25

Configures the maximum amount of times repair is attempted if the replica does not become consistent after repair completes, or repair fails because no consistent replicas are available to repair from.

repair.throttle.limit
10

Configures the maximum number of concurrent repair operations

repair.throttle.timeout
30

Configures a hard upper limit on how long repair will wait for a repair slot to become available

This value is in seconds. Using 0, or a negative value, disables the timeout completely. The default is 30 seconds.

repair.timeout.execution
86400

Configures a hard upper limit on how long repository repair is allowed to run.

This value is in seconds. Using 0, or a negative value, disables the timeout completely. The default is 24 hours.

repair.update-ref.batch-size
20000

Configures the maximum amount of refs to update in a single git update-ref operation as part of repair. If repair needs to update more than the configured limit, multiple git update-ref operations will be used.

Replication

Default valueDescription
replication.quorum-mode
majority

Controls how replica votes are applied when replicating transactions.

Available modes are: - always: If the leader prepares the transaction successfully, it is always committed--even if every other replica votes no. This mode makes write operations more likely to succeed, but may result in more frequent inconsistencies between nodes. - majority: Transactions are only committed if the majority (more than half) of replicas vote yes. Otherwise, the transaction is rolled back. Any replicas that voted no for a transaction that is still committed are marked as inconsistent and scheduled for repair. They will not process requests for the repository until repairs have completed. This mode balances consistency with availability. - unanimous: Transactions are only committed if every replica votes yes. Otherwise, the transaction is rolled back. This mode offers the best consistency, at the expense of making writes more fragile since a single replica failing will trigger a rollback.

The default is to require a majority vote.

Storage Management

Default valueDescription
storage.partition-cleanup-delay
3600

Determines the delay between when a topology update occurs and the storage cleanup job running. The cleanup job will delete partitions on disk that are not hosted by this Mesh node anymore. The default delay is 1 hour specified in seconds.

This value is in seconds.

storage.temp-cleanup-interval
24

Determines the interval used for how frequent the clean up of the tmp directory on the Mesh node job is run. The default interval is 1 day specified in hours.

storage.temp-expiry
72

Determines the interval used to calculate whether a file or directory is stale for it to be deleted on the Mesh node in the tmp directory. The default interval is 3 days specified in hours.

Throttling

These properties define concurrent task limits for the ThrottleService, limiting the number of concurrent operations of a given type that may be run at once. This is intended to help prevent overwhelming the server hardware with running processes. Two settings are used to control the number of processes that are allowed to process in parallel: one for the web UI and one for 'hosting' operations (pushing and pulling commits, cloning repositories).

When the limit is reached for the given resource, the request will wait until a currently running request has completed. If no request completes within a configurable timeout, the request will be rejected.

The underlying machine-level limits these are intended to prevent hitting are very OS- and hardware-dependent, so you may need to tune them for your instance. When hyperthreading is enabled for the server's CPUs, for example, it is likely that the default settings will allow sufficient concurrent operations to saturate the I/O on the machine. In such cases, we recommend starting off with a less aggressive default on multi-cored machines; the value can be increased later if hosting operations begin to back up.

Additional resource types may be configured by defining a key with the format throttle.resource.&lt;resource-name&gt;. When adding new types, it is strongly recommended to configure their ticket counts explicitly using this approach.

Default valueDescription
scaling.concurrency
cpu

Allows adjusting the scaling factor used. Various other CPU/throttling dependent properties are defined in terms of this setting, so adjusting this value implicitly adjusts those. Some examples:

  • event.dispatcher.core.threads

  • event.dispatcher.max.threads

  • executor.max.threads

  • throttle.resource.git-hosting

The default value, cpu, resolves to the number of detected CPU cores. On hyperthreaded machines, this will be double the amount of physical cores.

throttle.resource.git-lfs
80

Limits the number of Git LFS file transfer operations which may be running concurrently. This is primarily intended to prevent Git LFS requests from consuming all available connections thereby degrading UI and Git hosting operation. This should be a fraction of the maximum number of concurrent connections permitted by Tomcat.

throttle.resource.git-lfs.timeout
0

Controls how long threads will wait for Git LFS uploads/downloads to complete when the system is already running the maximum number of concurrent transfers. It is recommended this be set to zero (i.e. don't block) or a few seconds at most. Since waiters still hold a connection, a non-zero wait time defeats the purpose of this throttle.

This value is in seconds.

throttle.resource.mirror-hosting
2*${scaling.concurrency}

Limits the number of SCM hosting operations served to mirrors, which may be running concurrently. This limit is intended to protect the system's CPU and memory from being consumed excessively by mirror operations.

Note that dynamic throttling is not supported for mirror hosting operations.

throttle.resource.mirror-hosting.timeout
3600

Controls how long threads will wait for mirrors hosting operations to complete when the system is already running the maximum number of git commands.

This value is in seconds.

throttle.resource.scm-command
50

Limits the number of git commands, such as: git diff, git blame, or git rev-list, which may be running concurrently. This limit is intended to prevent the operations which support the UI from preventing push/pull operations from being run.

throttle.resource.scm-command.timeout
2

Controls how long threads will wait for SCM commands to complete when the system is already running the maximum number of SCM commands.

This value is in seconds.

throttle.resource.scm-hosting.timeout
300

Controls how long threads will wait for SCM hosting operations to complete when the system is already running the maximum number of SCM hosting operations.

This value is in seconds.

throttle.resource.scm-hosting.strategy
adaptive

Specifies the strategy for throttling SCM hosting operations. Possible values are 'adaptive' and 'fixed'.

If 'fixed' is specified, throttle.resource.scm-hosting.fixed.limit is used as the fixed upper limit on the number of concurrent hosting operations.

If 'adaptive' is specified, the maximum number of hosting operations will vary between throttle.resource.scm-hosting.adaptive.min and throttle.resource.scm-hosting.adaptive.max based on how many hosting operations the system believes the machine can support in its current state and given past performance.

If any configured adaptive throttling setting is invalid and reverts to a default but this conflicts with other correctly configured or default settings, the throttling strategy will revert to 'fixed'. E.g. this will occur if throttle.resource.scm-hosting.adaptive.min is set to the same value as throttle.resource.scm-hosting.adaptive.max

throttle.resource.scm-hosting.adaptive.limit.min
1*${scaling.concurrency}

When the adaptive strategy is enabled for throttling SCM hosting operations, this sets the lower limit on the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently.

Setting a lower limit provides a way to specify a minimum service level for SCM hosting operations regardless of what the adaptive throttling heuristic believes the machine can handle.

There is limited support for mathematical expressions; +,-,*,/ and () are supported.

throttle.resource.scm-hosting.adaptive.limit.max
4*${scaling.concurrency}

When the adaptive strategy is enabled for throttling SCM hosting operations, this sets the upper limit on the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently. This is intended primarily to prevent pulls, which can be very memory-intensive, from exhausting a server's resources. Note that if the machine does not have sufficient memory to support this default value or an explicitly configured value, a smaller value will be chosen on startup.

Adaptive throttling will vary the total tickets There is limited support for mathematical expressions; +,-,*,/ and () are supported.

throttle.resource.scm-hosting.adaptive.cpu.target
0.75

When the adaptive strategy is enabled for throttling SCM hosting operations, this sets the target CPU utilisation for the machine (across all processors) which the system takes into consideration when calculating the current throttling limit.

This value represents a trade-off: higher numbers may boost raw throughput of hosting operations, but at the expense of overall system responsiveness for all users. Increasing the target too high or too low brings diminishing returns.

This must be a value between 0.0 and 1.0 and is a percentage of the total available CPU power across all cores

throttle.resource.scm-hosting.fixed.limit
1.5*${scaling.concurrency}

When the fixed strategy is enabled for throttling SCM hosting operations, this limits the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently. This is intended primarily to prevent pulls, which can be very memory-intensive, from exhausting a server's resources. There is limited support for mathematical expressions; +,-,*,/ and () are supported.

throttle.resource.scm-refs
8*${scaling.concurrency}

Limits the number of ref. advertisement operations, which may be running concurrently. Those are throttled separately from clone operations as they are much more lightweight and much shorter so we can and should allow many more of them running concurrently.

throttle.resource.scm-refs.timeout
120

Controls how long threads will wait for ref. advertisement operations to complete when the system is already running the maximum number of ref. advertisement operations.

This value is in seconds.

gRPC

Default valueDescription
grpc.server.address
0.0.0.0

The address the gRPC services should bind to. The default value binds to all hosts.

grpc.server.port
7777

The port to use for the gRPC services

grpc.server.ssl.cert-chain-path

The location of the file containing the full certificate chain, when SSL/TLS is used. Default is empty, which means SSL is disabled. Setting a value for this property will only have an effect if grpc.server.ssl.private.key.path is also defined.

grpc.server.ssl.private-key-path

The location of the file containing the private key, when SSL/TLS is used. Default is empty, which means SSL is disabled. Setting a value for this property will only have an effect if grpc.server.ssl.chain.path is also defined.

Last modified on May 31, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.