Dell EMC DataIQ
Discover, understand and act on all unstructured data across your environment

Dataset Management:

Dealing with data: opportunities and challenges

The unrelenting growth of unstructured data represents an enormous opportunity for organizations looking to transform their business and win in this new digital economy. While unstructured data is often a source of innovation and competitive differentiation, harnessing it effectively can be challenging. In many organizations today, data is frequently distributed across diverse storage platforms, trapped in infrastructure ‘silos’ dedicated to specific applications and stored in both on-premises and cloud environments. Because of this disorganization, IT managers and business users are frequently unable to answer basic questions about their data such as: Where is my data? Is it in the cloud or one of our enterprise storage platforms? How can I enable my department to find the data they need on their own without burdening IT? How can our teams access data to collaborate more effectively on projects?

Take control of data with Dell EMC DataIQ

With DataIQ from Dell EMC, your organization can answer these questions posed and many more. DataIQ dataset management and insights software empowers organizations to discover, understand and act on unstructured data-file and object-across heterogeneous storage systems and the cloud. By providing rapid search functionality and a unified file system view of Dell EMC PowerScale, ECS, third-party and cloud storage, DataIQ allows enterprises to visualize all unstructured data through a single pane of glass, effectively breaking down siloes of trapped data.

For business users and content creators, DataIQ offers self-service capabilities to find, use and move files to the most appropriate storage tier (e.g. all-flash or archive). DataIQ does this via a database capable of indexing billions of files and folders that would otherwise be difficult to access without a consolidated global file system view. For IT administrators, DataIQ provides an ability to manage storage costs, rapidly locate files and accurately report on the usage of storage infrastructure.

Essentials

• Unified, single pane of glass view provides insights into file and object data in the data center and the cloud
• High speed search and indexing provides “visibility without access” to data
• Data mobility capabilities enable bi-directional movement between file and object storage
• Self-service features allow content owners to easily move data from high-performance file storage to an object archive
• Purpose-built, in-memory database supports optimized analytical performance
• Reporting features reveal the cost of dormant and redundant data with chargeback/showback views or cost recovery reports
• File visibility without data access ensures data security
• A unique plugin ecosystem gives users the power to act on DataIQ insights
• Designed to work with Dell EMC unstructured storage systems, PowerScale, ECS, PowerStore, etc. as well as other third-party platforms
• PowerScale cluster monitoring capabilities also available - See the Dell EMC DataIQ: Storage Monitoring document for additional details

Key Features and Capabilities

Single pane of glass: DataIQ eliminates challenges associated with data silos by proving a holistic view into heterogeneous storage platforms on-premises and in the cloud. A single pane of glass view into unstructured datasets provides users with file-centric insights into data and enables intuitive navigation.

High speed scan and index: DataIQ’s optimized, near real-time scan and high-speed file indexing delivers immediate project and user information. Powerful search capabilities across disparate storage platforms allows users to locate data in seconds, no matter where it resides.

Attribute tagging: DataIQ can ‘tag’ an attribute and use that tag to query millions of files across any storage system, enabling business users and IT to visualize data in a true business context and optimize costs within their storage environment.

Data movement: DataIQ simplifies data mobility with bi-directional movement between file and object storage. Business owners are empowered to use self-service archive capabilities to move files to the most appropriate storage tier either onpremises or in the cloud.

Reporting: With DataIQ, IT and storage admins gain a deeper understanding of their environment and can more efficiently manage storage costs. They can report on the true cost of dormant and redundant data and generate chargeback/showback views or cost recovery reports. IT can also report on storage usage by project or even individual users and determine what actions need to be taken (i.e. deleting duplicates or placing dark data under the proper governance policy).

The DataIQ plugin ecosystem inspires organizations to do more with data:

• Data mover: The data mover allows for a precision approach to transferring specifically selected files or folders from network filesystems to S3 or other targets. The plugin enables organizations to move data from on-premises storage platforms to the cloud and back again, ensuring projects and users have access to the right data, in the right place, at the right time.
• Duplicate finder: The duplicate finder plugin locates redundant data across volumes and folders, enabling users to delete duplicates, save costs and streamline their storage infrastructure.
• Audited delete: Audited delete helps IT and data managers keep a record of users who requested a delete. This provides visibility into who is actively deleting data from volumes so there is a ledger of actions taken in case critical data goes missing.
• Preview: The preview plugin provides DataIQ users with a thumbnail view of the specific file they’ve selected. The thumbnail is typically just a preview of whatever document, log, image, etc. is currently being inspected. This helps the user ascertain the context of the file without giving them full access to the data itself.

Note: Plugins are disabled by default and are optional to use.

Storage Monitoring:

Infrastructure optimization is key to effective data management

The extraordinary growth and management of unstructured data presents an enormous opportunity for organizations looking to transform their business and win in this new digital economy. But a key challenge underpinning an effective data strategy is managing the underlying storage platforms that support the massive data growth in the first place. Understanding the storage landscape and the critical applications it supports is of paramount importance for organizations who wish to ensure SLAs are met and TCO is optimized. Insights into performance bottlenecks, capacity thresholds being reached, network events, storage usage per workload or user, etc. are required to help IT tune the storage environment for the best results. Without this level of visibility, IT is essentially operating in the dark, resulting in poor system performance, mounting costs, and an inability to forecast future needs.

Powering up DataIQ with PowerScale monitoring functionality

Dell EMC’s DataIQ, a flagship dataset management and insights software, has introduced new storage monitoring functionality, all within the same tool. By including PowerScale OneFS cluster monitoring capabilities, IT can now manage both unstructured data and any PowerScale systems which support it, in a single, unified experience.

DataIQ’s advanced monitoring and reporting capabilities can help maximize the performance of your PowerScale systems. While OneFS and PowerScale enable your organization to easily manage petabytes of storage and computing resources, DataIQ takes it a step further by empowering IT to investigate performance bottlenecks within workflows, determine unstructured data growth trends, optimize resources with system health dashboards and forecast future capacity requirements.

DataIQ storage monitoring UI: Cluster Summary Dashboard

Essentials

• Achieve single-pane-of-glass visibility into PowerScale clusters
• Identify key warnings and critical cluster events
• Put all sites, including dark sites, under management
• Diagnose infrastructure, network and application bottlenecks
• Track OneFS jobs and services details-throughput, IOPS, CPU usage, cache hit rates, disk reads/write
• Understand past performance trends and predict future capacity requirements
• Leverage PowerScale cluster health insights to meet strict SLA requirements
• Monitor the PowerScale storage environment at enterprisescale-up to 70 clusters and 2000 nodes
• Supports PowerScale and Isilon clusters running OneFS 8.0+

PowerScale Monitoring Dashboards

DataIQ features a number of sub-dashboards and reports within the primary Cluster Summary Dashboard which displays the real-time status of each PowerScale cluster, alerting administrators to detrimental cluster events for further investigation and triage. Sub-dashboards include:

• Capacity Dashboard: Shows total capacity utilized per cluster over a period of time, empowering users to understand utilization rates and track capacity trends
• Network Dashboard: Displays network details pertaining to throughput, packet rate and error rate so IT can identify protocol and network latencies in order to tune PowerScale infrastructure for optimal results
• Client and User Dashboard: Presents information about protocol operations, protocol latency and network throughput, helping IT precisely pinpoint performance bottlenecks within clusters
• Filesystem Dashboard: Features shared directory details which pertain to file access deferment rates, helping identify users/clients consuming system resources (i.e. “noisy neighbors”)
• Hardware Dashboard: Presents a view of the OneFS cluster state from a hardware perspective, enabling organizations to discover usage and activity trends down to the granular node and disk level
• DR/Data Protection Dashboard: Provides summary information pertaining to OneFS SyncIQ policies, NDMP session events, ICAP and snapshot statuses to ensure replications jobs are performing as expected
• System Dashboard: Groups information about top jobs requiring attention (indicated by intensive resource consumption)
• Deduplication Dashboard: Displays an overview of deduplication and compression savings and efficiency across clusters
• Cluster Performance Report: Contains operations summaries from a cluster point of view, enabling granular drilldowns on protocol operation rates, latency, throughput, etc.

Documentation:

dell emc dataiq: dataset management datasheet (.pdf)

dell emc dataiq: storage monitoring datasheet (.pdf)