Warehouse FAQs

Can I control what data is sent to my warehouse?

Yes. Customers on Segment’s Business plan can choose which sources, collections, and properties sync to your data warehouse using Warehouse Selective Sync.

Selective Sync helps manage the data Segment sends to each warehouse, allowing you to sync different sets of data from the same source to different warehouses.

When you disable a source, collection or property, Segment no longer syncs data from that source. Segment won’t delete any historical data from your warehouse. When you re-enable a source, Segment syncs all events since the last sync. This doesn’t apply when a collection or property is re-enabled. Only new data generated after re-enabling a collection or property will sync to your warehouse.

Can we add, tweak, or delete some of the tables?

You have full admin access to your Segment Warehouse. However, don’t tweak or delete Segment generated tables, as this may cause problems for our systems that upload new data.

If you want to join across additional datasets, feel free to create and upload additional tables.

Can we transform or clean up old data to new formats or specs?

This is a common question if the data you’re collecting has evolved over time. For example, if you used to track the event Signup but now track Signed Up, you’d probably like to merge those two tables to make querying simple and understandable.

Segment does not have a way to update the event data in the context of your warehouse to retroactively merge the tables created from changed events. Instead, you can create a “materialized” view of the unioned events. This is supported in Redshift, Postgres, Snowflake, and others, but may not be available in all warehouses.

Protocols customers can also use Transformations to change events at the source, which applies to all cloud-mode destinations (destinations that receive data from the Segment servers) including your data warehouse. Protocols Transformations offer an excellent way to quickly resolve implementation mistakes and help transition events to a Segment spec.

Note: Transformations are currently limited to event, property and trait name changes, and do not apply to historical data.

How do I find my source slug?

Your source slug can be found in the URL when you’re looking at the source destinations page or live debugger. The URL structure will look like this:

https://segment.com/[my-workspace]/sources/[my-source-slug]/overview

How do I find my warehouse id?

Your warehouse id appears in the URL when you look at the warehouse destinations page. The URL structure looks like this:

​​app.segment.com/[my-workspace]/warehouses/[my-warehouse-id]/overview

How fresh is the data in Segment Warehouses?

Data is available in Warehouses within 24-48 hours. The underlying Redshift datastore has a subtle tradeoff between data freshness, robustness, and query speed. For the best experience, Segment needs to balance all three of these.

Real-time loading of the data into Segment Warehouses would cause significant performance degradation at query time because of the way Redshift uses large batches to optimize and compress columns. To optimize for your query speed, reliability, and robustness, Segment guarantees that your data will be available in Redshift within 24 hours.

As Segment improves and updates the ETL processes and optimizes for SQL query performance downstream, the actual load time will vary, but Segment ensures it’s always within 24 hours.

What if I want to add custom data to my warehouse?

You can freely load data into your Segment Warehouse to join against your source data tables.

The only restriction when loading your own data into your connected warehouse is that you should not add or remove tables within schemas generated by Segment for your sources. Those tables have a naming scheme of <source-slug>.<table> and should only be modified by Segment. Arbitrarily deleting columns from these tables may result in mismatches upon load.

If you want to insert custom data into your warehouse, create new schemas that are not associated with an existing source, since these may be deleted upon a reload of the Segment data in the cluster.

We highly recommend scripting any sort of additions of data you might have to warehouse, so that you aren’t doing one-off tasks that can be hard to recover from in the future in the case of hardware failure.

Which IPs should I whitelist?

You can whitelist Segment’s custom IP 52.25.130.38/32 while authorizing Segment to write in to your Redshift or Postgres port.

If you’re in the EU region, use CIDR 3.251.148.96/29.

BigQuery does not require whitelisting an IP address. To learn how to set up BigQuery, check out our set up guide

Will Segment sync my historical data?

We will automatically load up to 2 months of your historical data when you connect a warehouse.

For full historical backfills you’ll need to be a Segment Business plan customer. If you’d like to learn more about our Business plan and all the features that come with it, check out our pricing page.

What do you recommend for Postgres: Amazon or Heroku?

Heroku’s simple set up and administration process make it a great option to get up and running quickly.

Amazon’s service has some more powerful features and will be more cost-effective for most cases. However, first time users of Amazon Web Services (AWS) will likely need to spend some time with the documentation to get set up properly.

How do I prevent a source from syncing to some or all warehouses?

When you create a new source, the source syncs to all warehouse(s) in the workspace by default. You can prevent the source from syncing to some or all warehouses in the workspace in two ways:

• Segment app: When you add a source from the Workspace Overview page, deselect the warehouse(s) you don’t want the source to sync to as part of the “Add Source” process. All warehouses are automatically selected by default.
• Config API: Send a PATCH Connected Warehouse request to update the settings for the warehouse(s) you want to prevent from syncing.

After a source is created, you can enable or disable a warehouse sync within the Warehouse Settings page.