Official

Splunk destination integration documentation

Name: CloudQuery Splunk Integration
Author: CloudQuery

The Splunk destination plugin syncs data from any CloudQuery source plugin(s) to Splunk

Publisher

cloudquery

Latest version

v2.2.1

Type

Destination

Platforms

Date Published

Start Syncing

Overview Licenses

Overview #

Splunk Destination Plugin

The Splunk plugin syncs data from any CloudQuery source plugin(s) to Splunk.

Example config #

kind: destination
spec:
  name: splunk
  path: cloudquery/splunk
  registry: cloudquery
  version: "v2.2.1"
  write_mode: "append"
  spec:
    # Splunk Cloud configuration parameters
    host: "${SPLUNK_HOST}"
    port: 8088
    hec_token: "${SPLUNK_HEC_TOKEN}"
    
    # Optional parameters
    # protocol: https
    # rest_port: 8089
    # session_key: "${SPLUNK_SESSION_KEY}
    # index: main
    # event_source: cloudquery
    # event_sourcetype: _json
    # concurrency: 16
    # batch_size: 1000
    # batch_size_bytes: 5242880

The Splunk destination utilizes batching, and supports batch_size and batch_size_bytes.

It supports append mode only for write_mode.

Splunk Spec #

This is the spec used by the Splunk destination plugin.

host (string) (optional) (default: "localhost")
Host of the Splunk instance. Can be local or remote.
port (integer) (optional) (default: 8088)
Port of the Splunk instance.
hec_token (string) (required)
Splunk HEC token. Go to Settings -> Data Inputs -> HTTP Event Collector -> New Token to create a new token. https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector
protocol (string) (optional) (default: https)
Protocol for connecting to Splunk. Can be also specified in host.
rest_port (integer) (optional) (default: 8089)
REST API access port of the Splunk instance. Not used besides tests.
session_key (string) (optional)
Splunk session key used for rest API access. Not used besides test. Can be acquired for example by logging in using /services/auth/login Splunk endpoint
index (string) (optional) (default: main)
Splunk index to use for writing the events
event_source (string) (optional) (default: cloudquery)
The source field of events sent to Splunk will have this value
event_sourcetype (string) (optional) (default: _json)
Sourcetype field of events sent to Splunk will have this value
batch_size (integer) (optional) (default: 3000)
Maximum number of items in a batch that may be received by the plugin at a time.
batch_size_bytes (integer) (optional) (default: 15728640 (15 MiB))
Maximum size of items in a batch that may be received by the plugin at a time.
max_concurrent_requests (integer) (optional) (default: 3)
Maximum number of concurrent requests to the Splunk instance. Consider keeping the ratio batch_size / max_concurrent_requests around 1000, while keeping max_concurrent_requests as low as possible. Doing otherwise may result in increasingly high response times from the Splunk instance.
insecure_skip_verify_tls (boolean) (optional) (default: false)
Whether to skip verifying the TLS certificate of the Splunk instance. Useful for self-signed certificates or development environments.

Licenses #

The following tools / packages are used in this plugin:

Name	License
github.com/adrg/xdg	MIT
github.com/apache/arrow-go/v18	Apache-2.0
github.com/apache/arrow-go/v18	BSD-3-Clause
github.com/apache/arrow/go/v13	Apache-2.0
github.com/apache/arrow/go/v13	BSL-1.0
github.com/apache/arrow/go/v13	MIT
github.com/apache/arrow/go/v13	BSD-2-Clause
github.com/apache/arrow/go/v13	BSD-3-Clause
github.com/apache/arrow/go/v13	LicenseRef-C-Ares
github.com/apache/arrow/go/v13	OpenSSL
github.com/apache/arrow/go/v13	Zlib
github.com/apache/arrow/go/v13	NCSA
github.com/apapsch/go-jsonmerge/v2	MIT
github.com/aws/aws-sdk-go-v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/config	Apache-2.0
github.com/aws/aws-sdk-go-v2/credentials	Apache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsources	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight	BSD-3-Clause
github.com/aws/aws-sdk-go-v2/internal/v4a	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanager	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemetering	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/signin	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sso	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidc	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sts	Apache-2.0
github.com/aws/smithy-go	Apache-2.0
github.com/aws/smithy-go/internal/sync/singleflight	BSD-3-Clause
github.com/aws/smithy-go/transport/http/protocol/internal/json/internal/stdlib	BSD-3-Clause
github.com/bahlo/generic-list-go	BSD-3-Clause
github.com/buger/jsonparser	MIT
github.com/cenkalti/backoff/v5	MIT
github.com/cespare/xxhash/v2	MIT
github.com/cloudquery/cloudquery-api-go	MPL-2.0
github.com/cloudquery/codegen/jsonschema/docs	MPL-2.0
github.com/cloudquery/plugin-pb-go	MPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/glob	MIT
github.com/cloudquery/plugin-sdk/v2/schema	MIT
github.com/cloudquery/plugin-sdk/v2/types	MPL-2.0
github.com/cloudquery/plugin-sdk/v4	MPL-2.0
github.com/cloudquery/plugin-sdk/v4/glob	MIT
github.com/cloudquery/plugin-sdk/v4/scalar	MIT
github.com/davecgh/go-spew/spew	ISC
github.com/getsentry/sentry-go	MIT
github.com/ghodss/yaml	MIT
github.com/ghodss/yaml	BSD-3-Clause
github.com/go-logr/logr	Apache-2.0
github.com/go-logr/stdr	Apache-2.0
github.com/goccy/go-json	MIT
github.com/google/flatbuffers/go	Apache-2.0
github.com/google/uuid	BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors	Apache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2	BSD-3-Clause
github.com/hashicorp/go-cleanhttp	MPL-2.0
github.com/hashicorp/go-retryablehttp	MPL-2.0
github.com/invopop/jsonschema	MIT
github.com/klauspost/compress	MIT
github.com/klauspost/compress	Apache-2.0
github.com/klauspost/compress	BSD-3-Clause
github.com/klauspost/compress/internal/snapref	BSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhash	MIT
github.com/mattn/go-colorable	MIT
github.com/mattn/go-isatty	MIT
github.com/oapi-codegen/runtime	Apache-2.0
github.com/pb33f/ordered-map/v2	Apache-2.0
github.com/pierrec/lz4/v4	BSD-3-Clause
github.com/pmezard/go-difflib/difflib	BSD-3-Clause
github.com/rs/zerolog	MIT
github.com/samber/lo	MIT
github.com/santhosh-tekuri/jsonschema/v6	Apache-2.0
github.com/spf13/cobra	Apache-2.0
github.com/spf13/pflag	BSD-3-Clause
github.com/stoewer/go-strcase	MIT
github.com/stretchr/testify	MIT
github.com/thoas/go-funk	MIT
github.com/zeebo/xxh3	BSD-2-Clause
go.opentelemetry.io/auto/sdk	Apache-2.0
go.opentelemetry.io/otel	Apache-2.0
go.opentelemetry.io/otel	BSD-3-Clause
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	BSD-3-Clause
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	BSD-3-Clause
go.opentelemetry.io/otel/exporters/otlp/otlptrace	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace	BSD-3-Clause
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	BSD-3-Clause
go.opentelemetry.io/otel/log	Apache-2.0
go.opentelemetry.io/otel/log	BSD-3-Clause
go.opentelemetry.io/otel/metric	Apache-2.0
go.opentelemetry.io/otel/metric	BSD-3-Clause
go.opentelemetry.io/otel/sdk	Apache-2.0
go.opentelemetry.io/otel/sdk	BSD-3-Clause
go.opentelemetry.io/otel/sdk/log	Apache-2.0
go.opentelemetry.io/otel/sdk/log	BSD-3-Clause
go.opentelemetry.io/otel/sdk/metric	Apache-2.0
go.opentelemetry.io/otel/sdk/metric	BSD-3-Clause
go.opentelemetry.io/otel/trace	Apache-2.0
go.opentelemetry.io/otel/trace	BSD-3-Clause
go.opentelemetry.io/proto/otlp	Apache-2.0
go.yaml.in/yaml/v4	MIT
golang.org/x/exp	BSD-3-Clause
golang.org/x/net	BSD-3-Clause
golang.org/x/sync/errgroup	BSD-3-Clause
golang.org/x/sys	BSD-3-Clause
golang.org/x/text	BSD-3-Clause
golang.org/x/xerrors	BSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbody	Apache-2.0
google.golang.org/genproto/googleapis/rpc/status	Apache-2.0
google.golang.org/grpc	Apache-2.0
google.golang.org/protobuf	BSD-3-Clause
gopkg.in/yaml.v2	Apache-2.0
gopkg.in/yaml.v3	MIT

Loading plugin documentation

Overview #

Splunk Destination Plugin

The Splunk plugin syncs data from any CloudQuery source plugin(s) to Splunk.

Example config #

kind: destination
spec:
  name: splunk
  path: cloudquery/splunk
  registry: cloudquery
  version: "v2.2.1"
  write_mode: "append"
  spec:
    # Splunk Cloud configuration parameters
    host: "${SPLUNK_HOST}"
    port: 8088
    hec_token: "${SPLUNK_HEC_TOKEN}"
    
    # Optional parameters
    # protocol: https
    # rest_port: 8089
    # session_key: "${SPLUNK_SESSION_KEY}
    # index: main
    # event_source: cloudquery
    # event_sourcetype: _json
    # concurrency: 16
    # batch_size: 1000
    # batch_size_bytes: 5242880

The Splunk destination utilizes batching, and supports batch_size and batch_size_bytes.

It supports append mode only for write_mode.

Splunk Spec #

This is the spec used by the Splunk destination plugin.

host (string) (optional) (default: "localhost")
Host of the Splunk instance. Can be local or remote.
port (integer) (optional) (default: 8088)
Port of the Splunk instance.
hec_token (string) (required)
Splunk HEC token. Go to Settings -> Data Inputs -> HTTP Event Collector -> New Token to create a new token. https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector
protocol (string) (optional) (default: https)
Protocol for connecting to Splunk. Can be also specified in host.
rest_port (integer) (optional) (default: 8089)
REST API access port of the Splunk instance. Not used besides tests.
session_key (string) (optional)
Splunk session key used for rest API access. Not used besides test. Can be acquired for example by logging in using /services/auth/login Splunk endpoint
index (string) (optional) (default: main)
Splunk index to use for writing the events
event_source (string) (optional) (default: cloudquery)
The source field of events sent to Splunk will have this value
event_sourcetype (string) (optional) (default: _json)
Sourcetype field of events sent to Splunk will have this value
batch_size (integer) (optional) (default: 3000)
Maximum number of items in a batch that may be received by the plugin at a time.
batch_size_bytes (integer) (optional) (default: 15728640 (15 MiB))
Maximum size of items in a batch that may be received by the plugin at a time.
max_concurrent_requests (integer) (optional) (default: 3)
Maximum number of concurrent requests to the Splunk instance. Consider keeping the ratio batch_size / max_concurrent_requests around 1000, while keeping max_concurrent_requests as low as possible. Doing otherwise may result in increasingly high response times from the Splunk instance.
insecure_skip_verify_tls (boolean) (optional) (default: false)
Whether to skip verifying the TLS certificate of the Splunk instance. Useful for self-signed certificates or development environments.

Name

License

github.com/adrg/xdg

MIT

github.com/apache/arrow-go/v18

Apache-2.0

github.com/apache/arrow-go/v18

BSD-3-Clause