Good design? Smart pointers for allocation combined with raw pointers for access

I'm designing a FilterData class that reads in a custom binary data file and builds a data model of it in memory. I want to allocate all memory using smart pointers and let this FilterData have ownership of the data. When a caller needs access to the data, it will just get a raw pointer from the FilterData instance.

Do you think this is good design or would you find this "mixture" confusing as a programmer?

Some thoughts:

1. Why raw pointers? I need to interface with existing APIs that use raw float*. Furthermore I emphasize the caller does not own the allocated memory. Most of my "clients" are coders who are not so experienced in handling smart pointers.

2. If I ever need a copy of the data, I can extend FilterData with corresponding functions that return std::unique_ptr with a copy (therefore transferring ownership of the copy).

3. As a possible danger I see that someone could call delete on the returned pointer, but since I plan on doing all memory allocation with smart pointer, any delete would be erroneous anyway.

Here's some example code: (available here @repl.it)

#include <iostream>
#include <vector>
#include <string>
#include <memory>

class FilterData
{
  using fvec = std::vector<float>;
public:
  FilterData(const std::string& filename) {
    // (simulate read from file...)
    m_filterACoeffs = std::make_unique<fvec>();
    m_filterBCoeffs = std::make_unique<fvec>();
    m_filterACoeffs->emplace_back(1.f); m_filterACoeffs->emplace_back(2.f); m_filterACoeffs->emplace_back(3.f);
    m_filterBCoeffs->emplace_back(-1.f); m_filterBCoeffs->emplace_back(-2.f); m_filterBCoeffs->emplace_back(-3.f);
  }
  const fvec* getFilterACoeffs() { return m_filterACoeffs.get(); }
  const fvec* getFilterBCoeffs() { return m_filterBCoeffs.get(); }

private:
  std::unique_ptr<fvec> m_filterACoeffs;
  std::unique_ptr<fvec> m_filterBCoeffs;

};

int main() 
{
  FilterData filterData("thefile.bin");
  auto p_filterACoeffs = filterData.getFilterACoeffs();
  auto p_filterBCoeffs = filterData.getFilterBCoeffs();

  std::cout << "Filter A is: " << p_filterACoeffs->at(0) << " , " 
                               << p_filterACoeffs->at(1) << " , " 
                               << p_filterACoeffs->at(2) << std::endl; 

  std::cout << "Filter B is: " << p_filterBCoeffs->at(0) << " , " 
                               << p_filterBCoeffs->at(1) << " , " 
                               << p_filterBCoeffs->at(2) << std::endl; 
}