Why we care about the SharePoint Cloud Search Service Application
Earlier this month, I was lucky enough to attend Microsoft’s Ignite 2015 conference in Chicago along with a handful of other Content and Code colleagues. When we weren’t eating delicious (but oh-so-cheesy!) deep pan pizza, we were learning about upcoming Microsoft technology changes. Although Ignite focused on Microsoft’s public clouds, a few SharePoint Server 2016 sessions were scattered throughout the conference. We have found that there is still significant demand for on-premises SharePoint expertise, so I made sure to attend those sessions.
Today’s hybrid SharePoint Cloud search challenges
A stand-out session for me introduced the forthcoming Cloud Search Service Application, which looks to be the most significant enhancement to-date for the SharePoint hybrid story. It promises to overcome a major obstacle in the present SharePoint search hybrid story: today, there is no way to “merge” SharePoint Online and on-premises search results. Sure, you can “mash up” results using code (as documented by my colleague Chris O’Brien), but that doesn’t give you a single unified index and associated benefits (a consolidated relevance/ranking model, and one index to maintain).
While the absence of a merged index is probably the most common user-impacting complaint about today’s SharePoint hybrid search model, we have found through design discussions with our larger enterprise clients that maintaining the on-premises index required for hybrid search can be equally troublesome for SharePoint administrators. Firstly, all but the smallest SharePoint 2013 Search Service Applications require four servers, assuming you need high availability. Search servers are resource hungry, and patching them can be an onerous task (even if the correct procedures are followed). Secondly, it is a fact of life that full crawls must be run regularly in an on-premises SharePoint farm – TechNet lists more than a dozen reasons for doing so. Running a full crawl against a large and/or geographically distributed corpus might take days or even weeks. While a full crawl is running, changes to content are unlikely to be reflected in search queries, which will frustrate your users. There is also that nagging feeling that one day, your on-premises search index might need to be reset, rendering solutions that are dependent on search useless for the duration of the subsequent full crawl(s). Wouldn’t it be nice if you could make this someone else’s problem?
Enter the Cloud Search Service Application
The big idea is that Microsoft look after your search index in the cloud, regardless of whether its content was indexed on-premises or in SharePoint Online (SPO). Since there will be one unified index, your users will finally have a single set of results that can be ordered by relevance in an authentic manner.
Prior to Ignite, Microsoft referred to this thing as a “Hybrid Search Crawl Appliance“, so I was slightly surprised to learn that this new capability will be bundled in the Search Service Application. This means that you will still require an on-premises SharePoint Server 2013 or 2016 farm configured to crawl and parse on-premises content, and support hybrid authentication flows. The footprint of a farm designed to host a Cloud Search Service Application is likely to be a few shoe sizes smaller than a farm that might support today’s hybrid search scenarios, as fewer search components are required. To be more specific, the only on-premises search component strictly required by the Cloud Search Service Application is the crawler, and you have the option of leveraging a query processing component if queries must flow “outbound” from an on-premises farm to SPO. Note: all Search Service Application components (including the admin, analytics processing, content processing, and index components) were present within the on-premises SharePoint 2016 search topology during the Ignite demo. However, I understand that only the crawl and query processing components are actually used when a Search Service Application is “cloud enabled”, meaning that only those components would require performance and capacity planning in this context.
A slide from Microsoft Ignite illustrating how search roles are split between on-premises SharePoint and SPO when using the Cloud Search Service Application.
Other than crawling, the jobs performed by other search components (content processing, ACL mapping, index building etc.) are outsourced to SharePoint Online, meaning that you no longer have to maintain an on-premises search index. A copy of parsed content is stored in SPO so that cloud-based search infrastructure changes do not require on-premises content to be re-indexed. It remains to be seen precisely how much easier a cloud-enabled Search Service Application will be to manage, but I expect some of the administration problems I’ve highlighted in this blog will go away. At the very least, I think that index resets will become a thing of the past. The optimist in me thinks that some of the other reasons to run a full crawl in SharePoint 2013 today will also become non-issues (SharePoint Online runs on something that closely resembles a more recent version of whatever binaries are available on-premises, so patching my SharePoint farm shouldn’t require a full crawl. Right?).
A slimmer on-premises SharePoint farm
During the Ignite session, Microsoft gave an example that indicated a company might need 10 on-premises SharePoint 2013 servers to support a “traditional” Search Service Application. A separate slide indicated that as few as 2 servers would support the Cloud Search Service Application:
A slide from Microsoft Ignite indicating that the Cloud Search Service Application may only require 2 SharePoint Server 2013 servers.
It was unclear whether that example included any servers required for SQL Server, and it raises other questions (will the Distributed Cache Service happily co-exist with the Cloud Search Service Application?), so I’ll wait for more detailed guidance before getting too excited.
Give me more detail!
I know many readers out there will be technical, so I’ve included the bullet point notes I took during the session below. Keep in mind that we got a pretty early view of the capability at Microsoft Ignite, so these details are likely to change between now and General Availability.
The Cloud Search Service Application:
- Will be shipped in an update to SP2013 later this year (2015), and will be baked into SP2016.
- Pushes indexed items into a single consolidated SPO index, instead of relying on query federation (the current SP2013 search hybrid model). This means we get a single ranked results set with refiners and search previews. About time!
- Means that on-premises content shows up in Delve (which uses the SPO search index), albeit without the “rich” thumbnail previews that we get with SPO content
- Is able to crawl the same content sources that the present SP2013 Search Service application can crawl. This is great news, as it means that content housed in older (2007 or 2010) SharePoint farms can be surfaced in SPO, along with other supported content sources such as file shares
- Can be consumed by a *SharePoint 2010* farm using today’s SharePoint Service Application Federation model, meaning that SPO content can be queried from a SP2010 farm! As you might expect, there are a few trade-offs and constraints here (e.g. No WAC previews; Web Applications must be in claims mode, on-premises SP2013 Query Processing Component needed). This means that older farms need not be islands of information in a hybrid scenario
- Can co-exist with other Cloud Search Service Applications to feed a single SPO tenant from multiple locations. I expect that this will be a big deal for some of our globally distributed clients.
- Strictly respects on-premises permissions. SPO permissions do not “override” on-premises ACLs, even if you are an Office 365 Global Admin.
- Means that there is no longer a need to have an on-premises search index (save for data residency concerns, see next bullet point).
- Can co-exist with the current SP2013 search hybrid model (query federation) if some indexed items need to remain on-premises (e.g. data residency reasons)
- Will be baked into the SharePoint 2016 Search Service Application. This means we still need a “lightweight” farm to house it, complete with a SQL Server instance. I expect the update for SharePoint 2013 will work in the same way, although that wasn’t clarified.
- Relies on the same foundational hybrid identity management bits that are needed for todays’ SP2013 hybrid solutions (directory synchronisation, OAuth 2.0 trust between SharePoint on-premises and Azure ACS etc.)
- Relies on an on-premises Office Web Apps farm if Search Previews of on-premises SP2013 or SP2016 content are required (e.g. from within SPO search results pages). This is the only reason that an “inbound” (SPO -> on-premises) hybrid configuration would be required
- Does not “publish” on-premises content externally. We still need a user-facing publishing capability – such as the Web Application Proxy (WAP) or Azure App Proxy – for secure external publishing of SharePoint and/or Office Web Apps. The alternative is that on-premises content can be searched in SPO, but only accessed and previewed on-premises.
- Encrypts search metadata before it is sent to SPO in batches
- Does not support on-premises Site Collection-scoped schema mappings, as those Site Collection objects do not exist in SPO.
- Introduces a new Managed Property (IsExternalContent) that allows on-premises content to be identified in query rules, result sources, verticals etc.
As you can probably tell, we are pretty excited about getting our hands on this thing! In our experience, search is the most common reason for implementing a hybrid SharePoint infrastructure, and we are pleased to see that Microsoft are addressing the most common pain points about this workload. We will still require an on-premises SharePoint farm to achieve hybrid search, but hopefully it will meet user expectations, result in fewer sleepless nights for SharePoint administrators, and won’t break the bank.
That’s all from me, but you can watch the Ignite session for yourself over on Channel 9: “Implementing Next Generation SharePoint Hybrid Search with the Cloud Search Service Application“.
Head of SharePoint Platform
Ben leads Content and Code's SharePoint Platform practice which focuses on the more technical aspects of SharePoint Online, SharePoint on-premises and everything in between. He has been working with SharePoint and related technologies such as SQL Server and AD FS since 2008.
Other Services you may be interested in.
In the last post we covered off the results of an internal review of the features of Workplace by Facebook. In this post we will cover the value that organisations can gain from implementing a corporate social network like Workplace or Yammer. Then in the next blog post, we will discuss other tools that are required to compliment the solutions, in order for organisations to provide a full digital workplace solution.
So, why implement an enterprise social network like Workplace or Yammer?
It’s simple, email is a place where information goes to die. Email is locked to a specific person and works OK as an individual’s knowledge repository, but as such it is not designed to be an all-encompassing enterprise knowledge repository.
You can only search from within your own email client and if you were not on an email thread, then you have no access to that information. Ultimately email is great for personal messages but not for sharing and dissemination of corporate knowledge.
This is one of the key reasons organisations make the same mistake over and over. Essentially they have corporate amnesia, as instructions and lessons learned from previous projects are lost in endless email and any new staff never see this information and therefore repeat mistakes.
It is also the reason that departments get asked the same question over and over again. I have seen people copy and paste the same email response many times a day. I simply cannot fathom why people do not seek a better (and more modern) way to work.
There is a real cost to this lack of information sharing. In most organisations, people ask a colleague for information, they refer them to another person who generally refers them onto yet another person. The third or fourth person in the chain can typically help although a lot of time and effort has been wasted in the process.
This is a real cost to businesses.
McKinsey and IDC research uncovered that the average office worker spends one whole day every week searching for information. Let me put that into context for you.
If you had 1,000 staff (earning an average £35k salary) the cost in salaries alone for the time spent searching for information is £2.45million per year. McKinsey and IDC highlight that a corporate social network, built through social technologies, can improve the productivity of employees by 20% to 25% – in real terms that’s one whole day a week an employee could save.
That is like everyone working six days a week, rather than five. How much could that benefit your organisation? In crude money terms, you could save over £2milion in salaries a year, or save employees over 8000 hours per week!
Workplace or Yammer changes the game here.
Within an enterprise social network, messages are kept and questions and answers are searchable so any new staff or those not on the original thread can find this information. This is known as ‘working out loud’ – if the topic being discussed is not sensitive, then why do you need to have it locked down in email?
Another major benefit of working out loud is that staff are keen to learn more about other areas of the business and can contribute in meaningful ways. Naturally people want to help and if they can provide assistance or commentary they will.
This helps achieve both organisational goals and increases employee engagement as staff feel more connected and involved with the organisation.
Corporate social networks tend to have groups for departments, and also groups that act as communities of interest and communities of practice. This is where information can be shared openly and colleagues can contribute and learn from one another.
Having this open communication and knowledge sharing greatly increases employee engagement as staff can easily post their thoughts and ideas and ‘have a voice’. Employees naturally want to contribute and to help an organisation flourish, although unless the right platforms are in place for idea sharing and collective problem solving, then the people at the ‘coal face’ who can help solve issues and fix problems, often have no way of getting their opinion heard.
Some great examples of this in action are when a large brewery implemented Yammer and one of the truck loaders shared his ideas about how they could load trucks more efficiently by changing the box orientation on the pallets.
This saved the Company over £3 million per year – although when the CFO asked why the chap hadn’t spoken up before, he said he had told his old line manger many years ago although nothing changed so he didn’t bother following up.
The corporate social network gave him a voice where his ideas could be shared, properly aired and debated with a wider audience. Drawing on our own recent experience, we implemented Yammer for Canon, and as a result there were over 20 examples of direct business success resulting from Yammer within months of launch.
The other key benefit I see from implementing Yammer or Workplace is that organisations are able to be more responsive.
Business is becoming increasingly more fast paced and the companies who can react faster, innovate more rapidly and get things done quicker will be the leaders in their sectors.
A well adopted enterprise social network is like having all of your staff located in one big room, where everyone can easily listen in and contribute to relevant conversations. It massively increases the speed and fluidity of communications and makes organisations feel smaller and more friendly.
Essentially, things just happen faster. You get more staff buy-in when ideas are debated or problems discussed on the social network.
For example, we use Yammer at Content and Code: we use it for all our project updates, client stories, business development and marketing. Even sharing and gathering feedback on our monthly reporting is undertaken on Yammer – we are a more connected business because of it.
The social aspect of Yammer massively helps our employee engagement: we use the ‘praise’ functionality to recognise individuals and teams for their great work – which feeds into our star of the month awards.
Benefits of engaged employees
I’ve talked about direct business success and how a social network can help organisations, although let’s consider the benefits of having more engaged employees.
Gallup state that engaged employees are less likely to leave and also contribute more to their jobs, generally going outside of their job description to help. (Take a look at the full report here.)
So what does this mean in real money?
If you had 1,000 staff (earning an average £35k salary and with an industry average attrition rate of 15%) than you would lose and need to replace 150 staff every year. If you were efficient and had quite low costs of only £2,500 to find a new staff member and it took £2,500 to on-board and train them, this would cost your business £750,000 every year.
Gallup state that this can be reduced by up to 51% – that’s a saving of over £350,000 per year, or you would not have the aggravation of needing to find and replace 77 employees every year.
Finally, in terms of employees contributing more if they are engaged, then Gallup ‘s research shows that companies that have engaged employees have up to 27% higher profits than their peers. That is something I am sure any Finance Director, Managing Director or shareholder would be very interested in.
Look out for my next post where I will discuss other tools that most organisations need to provide a full digital workplace solution.
Embrace modern ways of working with an enterprise social network
If you are thinking about enterprise social networks for your workplace, sign up for our free morning event as we take a comparative look at Workplace by Facebook and Office 365 Enterprise.
About our author
CEO and Founder
Tim is the CEO of Content and Code and founded the company in 2001. As CEO of Content and Code, Tim focuses on corporate strategy, client and partner relationships, and has overall responsibility for driving the growth of the company. Tim founded Content and Code so he could help transform organisations to be more responsive, more competitive and engage their employees to better realise business goals.
If you are running older versions of SharePoint, there can be many reasons why your SharePoint environment may not be working or performing to full capacity. From Timer job failures to SharePoint Search problems, these issues can cause an onslaught of complaints from end-users, who are frustrated about SharePoint’s slow performance, and increased frustration of the SharePoint Admin, who are constantly using up time and resource to fix seemingly basic issues.
In this post, we’ll outline the top five SharePoint environment problems that we have encountered, and provide some quick tips on how to resolve, or prevent them from happening in the future.
Timer job failures
SharePoint 2013 has 155 out of the box timer jobs which keep a SharePoint environment functioning and performing. On top of that, many custom SharePoint solutions add additional timer jobs to the list. However, quite often, these timer jobs are not monitored or checked at all. As timer jobs run at the background, users won’t notice any issue initially until it impacts SharePoint applications.
Recently we helped one of our clients resolve an interesting issue. The timer job of the “Microsoft SharePoint Foundation Usage Data Import” failed to run. With insufficient SharePoint knowledge within their IT support team, no investigation was carried out. As a result, the usage logs kept growing on their web servers until it filled up the disk drive with more than 60GB logs, and the SharePoint application stopped functioning which caused unnecessary operational and IT issues.
To resolve the issue we restarted the SharePoint Timer Service, and the timer job of “Microsoft SharePoint Foundation Usage Data Import” started to work, which released more than 50GB logs from the disk drive.
SharePoint Search issues
SharePoint Search is one of the key functionalities within all versions of SharePoint. Nowadays, the majority of SharePoint applications are built upon the SharePoint search feature to surface data to the home page or landing page for performance benefits. This makes it even more important to make sure the SharePoint Search related services are working as expected within your SharePoint farm. Otherwise, users may use the out of date data without noticing.
In many cases that we come across, SharePoint Search issues only lie in a specific area of the SharePoint site. Although the search query does seem to surface the recently updated content, sometimes results will exclude content from a specific subsite.
How do you detect SharePoint Search issues? Well, quite often end users or administrators just assume the search works across all sites and that anything not in the search result simply doesn’t exist. This is not the case.
In a recent example, we helped a client who was experiencing issues with their Index Engine that was stuck on the crawling status for more than a week. Quite often the SharePoint administrator is the last person to find out the issue until the end users raise a ticket with their IT support department.
User Profile Sync service
User profile sync is another key functionality in SharePoint and quite often, it’s linked with the Active Directory service.
We had a client recently that configured their User Profile to sync back to Active Directory as they saw SharePoint the source of truth for user details and wanted the users to be responsible for updating their profile. In taking this step, the changes are automatically written back to Active Directory properties and in doing so it saves IT administrator’s the time of updating Active Directory user details manually.
It sounded a good idea until the sync service stopped working and they tried to restore the backup database, guess what? The old profile data were synced to their Active Directory. Painful, isn’t it?
The general lack of understanding the limitations or boundaries of SharePoint features can sometimes lead to the failure of the SharePoint application.
The application generally works well with a small set of users and contents. To draw on a recent piece of work undertaken by the SharePoint support team for a client, we found that as time went on, more and more content was created on the site and the application started struggling to perform. At this point we took a deeper look into the issue.
An obvious issue here is the 5k limit on a SharePoint list view. If more than 5k items were added to the list, it would throw an error when a user tried to browse the items in the list with the default view. To overcome the limitation, indexes need to be applied to certain columns and the list view may have to be re-designed.
What about the List View Lookup Threshold you ask? The client had managed to break all sorts of thresholds in a single list. SharePoint Lists were treated as database tables therefore, more and more lookup fields were added to the application. In fact, to combat this issue, some of the Lookup fields could easily be replaced by Choice fields.
In an ideal world, being able to identify which thresholds are approaching is the key to the success of your SharePoint farm, which gives you sufficient time to implement a fix or a different design.
The current version of SharePoint is no longer suitable for the demand
It’s hard to forecast the end-user demand of SharePoint accurately at the time of initial roll-out, and to be perfectly honest, it’s always going to be difficult to satisfy every user. But, the important thing here is that you need to monitor your system and carry out a regular SharePoint health check to ensure your environment is meeting end-user demands in an ever-evolving workplace.
Some clients we work with seem reluctant to invest time and resource into looking over the performance of their SharePoint systems. Ultimately, they end up investing more time and effort to fix the issues later on, and that’s not including the damage and impact that system problems have on lost productivity within the workforce.
SharePoint patching is a typical example as Microsoft releases security patches or other fixes frequently that are often overlooked.
We have had clients who have spent lots of time and internal resource to fix an issue, only to find out the issue has been addressed in the security patches. Not factoring in these patches, means that the SharePoint farm is running at risk of being exposed to a widely-known issue that is impacting that particular version of SharePoint.
Don’t forget, Microsoft support ceases to exist on 10th October 2017
You might be still running on SharePoint 2007 and thinking it is still working. Yes, it might be OK for now, but you could encounter an unfixable issue when Microsoft support runs out.
As with any major patching or upgrade to SharePoint – it is not a quick job. It requires an in-depth analysis of your SharePoint farm and meticulous planning to make sure changes are executed with minimal impact to your business. Remember that Microsoft’s extended support SharePoint 2007 ends on 10th October, 2017.
Ultimately, you don’t want to leave this until the last day. Speak to Content and Code today, and we can undertake a SharePoint health check for you for free.
Is your SharePoint environment performing as it should?
Make sure your SharePoint environment is performing to optimum capacity with a FREE SharePoint assessment
About our author
Technical Account Manager | Content and Code
Michael leads the DevOps team and is responsible for our Development On-Demand service prominently for SharePoint in Office 365, or on-premises. Michael has worked on a vast number of DevOps projects for clients large and small, often with complex and dynamic requirements.
Continuing our series on the changes that have taken place across the SharePoint platform in the last ten years, today we look at some of the important questions you need to ask of your SharePoint strategy.
Our question today is; should you buy your SharePoint environment outright, or instead rent out your access to the tool? Each approach has its own appeal, and your eventual choice will depend on your specific situation. We know both the challenges companies face when making this decision and the opportunities a new approach can bring. Let’s take a look at which approach is best for you.
First, a bit of background
Online, subscription-based IT services are increasingly the popular choice for businesses; from small start-ups to more established enterprise companies who previously would have invested heavily in on-premises IT services. One of the major reasons for this shift has been the IT industry’s move towards consistent, rolling updates rather than the bi-yearly releases of the past. Call it a rolling-release or continuous deployment; the idea is to provide numerous, regular updates and patches to users as they are generated. The argument for and against has existed for a long time, broken down into this comparison:
Do you want the very latest, newest improvements with potentially some glitches yet to be ironed out? Or the more consistent, ‘final’ version of the software/solution/platform?
This has up until recently been a legitimate question that businesses were required to consider. However, with Microsoft’s mobile-first, cloud-first ethos, this question is becoming ever more redundant. Essentially, the latest updates for Microsoft’s platforms and solutions are better and more consistently delivered without glitches or bugs, and it’s set to continue in this vein. The more comfortable we all get in the cloud, the more accepted and expected it will be for our tools and solutions to be updated on a rolling basis.
Secure your SharePoint environment with a successful SharePoint migration
SharePoint Online versus SharePoint on-premises
Perhaps the best example of this move towards subscription-based services is Microsoft’s SharePoint. A traditionally on-premises titan, Microsoft is now developing the platform with a foundation in the cloud. There will, of course, be new versions of SharePoint Server available (to accommodate those organisations still wanting an on-premises option), but the future, it seems, is in the cloud. Microsoft offers various monthly subscription prices that are dependent on the type of enterprise package that a customer chooses, and in our post today, we’ll discuss the cost-effectiveness of a subscription-based SharePoint iteration in the cloud (complete with rolling updates) versus a solely on-premises version.
The opportunity to change
If you didn’t know it already, SharePoint 2007 goes out of support by October, 2017. So if your business is one of the many that still uses this iteration, now is the time to begin thinking about what you want to do next. For many that means asking the following: should you continue to use SharePoint on-premises or should you make the leap to the cloud?
There are good reasons on both sides of the debate. A lot of companies who currently use on-premises are heavily invested in their IT infrastructure. You are likely to have a team dedicated to maintaining the SharePoint farm and the various servers, providing updates and patches. The good news for on-premises users is that downtime for updates has been largely removed.
Of course, the online iteration of SharePoint 2016 is updated automatically by Microsoft, removing downtime entirely as well as taking infrastructure costs out of the equation. Indeed, one of the big selling points of Office 365 is the financially-backed SLA of 99.9% uptime – organisations who don’t receive this level of service are eligible for refunds, in the form of service credits.
The opportunity to save
When we talk about on-premises versus online in any instance, one of the big talking points is the difference in resources needed for both. On-premises solutions require an entire backroom infrastructure, hardware, physical space and skilled personnel. Conversely, an online iteration, in the case of SharePoint, requires none of these physical entities (people, hardware, space) but rather monthly or annual subscriptions, and therefore significant savings can be made.
The change from on-premises to online is actually a change from capital expenditure (CAPEX) to operating expenses (OPEX). This is a noteworthy change to take into consideration – and one of the major reasons why smaller companies are able to run their businesses on a fraction of a larger organisation’s budget and yet still remain competitive.
On this point alone there are big savings to be made: for between £3.10-£6.20 per user per month you could set your company up with the very latest iteration of the SharePoint platform and all it entails. For £14.70 per user per month you can do the same and also get the very latest tools from Office 365 as well.
So what if you’re already heavily invested in an on-premises iteration? It’s a legitimate question for a lot of organisations out there, and that’s where hybrid comes into play. With hybrid, you can leverage all the best things about SharePoint in the cloud with Office 365, while consolidating (financially or for compliance reasons) what you’ve already committed to structurally. So, if you like the sound of SharePoint Online, but have a lot invested in on-premises, hybrid might be a great option for you. The latest release of SharePoint Server 2016 has seen Microsoft put in a great deal of effort into the hybrid functionalities of the platform.
That’s not all
So far in this series we have covered a number of features that are offered by the newer versions of SharePoint. What’s more, by migrating to SharePoint in the cloud you can get all the benefits of these features at a cheaper price than you are currently spending.
If you’re still unsure of your SharePoint future, or if you want to find out more about how a migration might be a great fit for your business, sign up for our webinar “Steps to completing a successful SharePoint migration“.
Secure your SharePoint future
Learn the key steps to completing a successful SharePoint migration
About our guest author
Partner Solutions Engineer | Metalogix
Chris Thorpe is the partner solutions engineer at Metalogix, who are experts in providing specialist solutions for SharePoint migrations. With a plethora of experience in demonstrating solutions and training clients, mapping out a successful migration is Chris’s bread and butter.